Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....X] duration: 5.001629664s trace: https://jaeger.dev.storj.io/trace/786e0102014f212f FAIL (2025-12-30 11:52:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001651818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X...] duration: 6.011147371s trace: https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426 FAIL (2025-12-30 11:51:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments... Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment... Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment. Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments... Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces. Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments... Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil> Duration: 6.011228215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X...........X....X.X........X................X] duration: 1m0.003743641s trace: https://jaeger.dev.storj.io/trace/1f1c46e5c6b81134 FAIL (2025-12-30 11:51:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 1m57.317198375s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446504321 (596h21m57.317587356s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.446947225 (596h21m57.31803026s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447416789 (596h21m57.318499854s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.447814737 (596h21m57.318897772s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448193398 (596h21m57.319276433s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448580755 (596h21m57.31966379s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.448588319 (596h21m57.319671354s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449032025 (596h21m57.32011507s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449433820 (596h21m57.320516855s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.449966063 (596h21m57.321049098s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450396162 (596h21m57.321479197s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.450663060 (596h21m57.321746106s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451135239 (596h21m57.322218274s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.451694094 (596h21m57.322777129s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:57.453724099 (596h21m57.324807144s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Uploading piece... Dec 30 11:50:58.471603816 (596h21m58.342686881s) [noise=true , address=193.149.164.143:4700 , piece_id=7DYYZ4DEB3LZ4B5P3HOQ2A4XSAGOHH7VAL6SZQMSQGKZHGSCMTJA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 30 11:50:58.814556042 (596h21m58.685639087s) [upload=522768]: Next returned nil segment Dec 30 11:50:58.814569308 (596h21m58.685652353s) [upload=522768]: Waiting for error group managing segments... Dec 30 11:50:59.307376261 (596h21m59.178459296s) [noise=true , address=145.79.227.2:4124 , piece_id=5DSIBHDVYUU3FOJCBT354HQ56BHEBXFYQPVSKISZ6HBZYJR4C25Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.314712290 (596h21m59.185795325s) [noise=true , address=145.79.227.4:4211 , piece_id=76TKCGXGX4MAOAUBXI6EB2SKT5BGMQIQAS6KXNPYKESWKPNXYZPA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.316343256 (596h21m59.187426301s) [noise=true , address=145.79.227.18:4023 , piece_id=L6JU2KFXHCTDBZE6WOO4YCB64ORWNTHBVAO55YG5XXDJLPBSNRNQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.366460096 (596h21m59.237543131s) [noise=true , address=145.79.227.13:4824 , piece_id=SM3VMEQG6CVYAMQJKI7S5B3FTT544HGAPLD2BH3ANPUPZDKY37NA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.507706272 (596h21m59.378789317s) [noise=true , address=77.90.31.21:4897 , piece_id=IVYKWW2D3K46P6JOCLQV5TO3B4WC3XWRHCDZMX7AJVLFDGW4ZSOQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.596487946 (596h21m59.467570991s) [noise=true , address=77.90.31.15:4717 , piece_id=5UXWTB6TFVYRQOXSN7JUNUZL37YVW4YRMBLZF2RGEANROQLWPGMQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.666816143 (596h21m59.537899188s) [noise=true , address=77.90.31.19:4865 , piece_id=J7XQIT7OAPSXYJAMKH4YS6ZVBSDPYKN4JWIQYHJHTXJKMURNOPYQ, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.717575376 (596h21m59.588658411s) [noise=true , address=77.90.31.10:4532 , piece_id=UHT6PIQFSSNHUNNADAC5Y7Q3OHA6NCCYTB5T62VBCOMCB2IABQZA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:50:59.771747729 (596h21m59.642830774s) [noise=true , address=45.91.116.13:4170 , piece_id=5DY5YELTKCNT3YIWSWUPLZIJZ2CNEY5R7GYNCDWPRYX3P2TO7TYA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.238826642 (596h22m0.109909687s) [noise=true , address=145.79.227.44:4196 , piece_id=BLS62U5EMVWG7RY2YZBNSQ2VGIWCZPWGCZPVEB36QAEE2KQ5PKNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.255935080 (596h22m0.127018115s) [noise=true , address=145.79.227.34:4925 , piece_id=P7UTUSY5TTMGMLYDPIHHX3RGZFDDGVEODOV3NXFY5KRM3OJWVESA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.267803509 (596h22m0.138886554s) [noise=true , address=145.79.227.30:4771 , piece_id=B5ZTKVL4HYJGASKQPMSPVVRJ2AS5EVNIQLVSMT26P3XNSSH4ADMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.274189218 (596h22m0.145272263s) [noise=true , address=145.79.227.33:4899 , piece_id=VMIOQVODV2EFQ33RK36YCJJBQVOTYJREN3U2V55FYXTKCR7XBRUA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.308979524 (596h22m0.180062569s) [noise=true , address=64.49.13.27:4964 , piece_id=ISAY3PSAY7O3FJVIO63QE5WBG76FUASY72QETA4JHDPSXMQWVIMA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: <nil> Dec 30 11:51:00.309006346 (596h22m0.180089391s) [seg_pos={0 0}, upload=522768]: Segment reached optimal threshold of 14 pieces. Dec 30 11:51:00.309236043 (596h22m0.180319088s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46640->77.90.31.18:4838: use of closed network connection Dec 30 11:51:00.309261382 (596h22m0.180344417s) [noise=true , address=77.90.31.18:4838 , piece_id=AN5HDI2HJAR5CN7ZTHNT5IICZHQTDAYC2LSYVC54OZYP37JZL7WA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309474938 (596h22m0.180557984s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44408->64.49.13.169:4715: use of closed network connection Dec 30 11:51:00.309490808 (596h22m0.180573843s) [noise=true , address=64.49.13.169:4715 , piece_id=QJM2GC2MKZGVSJGW3L57NW56C4P7JOMRNKSUKMMTZIEOIADYMG4Q, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309622259 (596h22m0.180705304s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 30 11:51:00.309631216 (596h22m0.180714261s) [noise=true , address=77.90.31.14:4671 , piece_id=GZFSL67SGTAQSJAHV7LY44LWHTRKDNYMV6MDOJON6Q5IDJMLVSNA, seg_pos={0 0}, upload=522768]: Long tail cancellation due to optimal threshold Dec 30 11:51:00.309654170 (596h22m0.180737275s) [seg_pos={0 0}, upload=522768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:51:00.309744170 (596h22m0.180827205s) [upload=522768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:51:00.309750163 (596h22m0.180833198s) [upload=522768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:51:00.603342559 (596h22m0.474425604s) [upload=522768]: Done uploading segments... Dec 30 11:51:00.603368449 (596h22m0.474451494s) [upload=522768]: Upload finished. err: <nil> Duration: 1m0.003767184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X..] duration: 26.836734833s trace: https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426 FAIL (2025-12-30 11:51:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments... Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment... Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment. Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments... Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces. Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments... Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil> Duration: 26.836758428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X....] duration: 4.725355215s trace: https://jaeger.dev.storj.io/trace/6e904e080c2bce20 FAIL (2025-12-30 11:51:01 UTC) orbiter warning: context deadline exceeded Duration: 4.725387085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X.] duration: 6.655171003s trace: https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426 FAIL (2025-12-30 11:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments... Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment... Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment. Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments... Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces. Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments... Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil> Duration: 6.655192244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.X] duration: 30.003760995s trace: https://jaeger.dev.storj.io/trace/2cb9ebc1aae8e426 FAIL (2025-12-30 11:50:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 11:49:55.778917620 (596h20m55.650000655s) [upload=522750]: Starting upload Dec 30 11:49:55.778969179 (596h20m55.650052214s) [upload=522750]: Done starting upload Dec 30 11:49:55.779087253 (596h20m55.650170298s) [upload=522750]: Uploading segments... Dec 30 11:49:55.779227650 (596h20m55.650310695s) [upload=522750]: Got next segment. Inline: false Dec 30 11:49:55.779260703 (596h20m55.650343738s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:49:55.779265442 (596h20m55.650348477s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:49:56.018714561 (596h20m55.889797606s) [seg_pos={0 0}, upload=522750]: Begin upload segment... Dec 30 11:49:56.018760468 (596h20m55.889843503s) [seg_pos={0 0}, upload=522750]: Done begin upload segment. Dec 30 11:49:56.018786117 (596h20m55.889869152s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.018839359 (596h20m55.889922394s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019310416 (596h20m55.890393451s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019616709 (596h20m55.890699744s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.019934945 (596h20m55.89101798s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020285953 (596h20m55.891368998s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.020769073 (596h20m55.891852108s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021175067 (596h20m55.892258102s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021310955 (596h20m55.892393991s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021670650 (596h20m55.892753695s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.021750812 (596h20m55.892833847s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022212091 (596h20m55.893295137s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.022637762 (596h20m55.893720797s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023001294 (596h20m55.894084329s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.023519691 (596h20m55.894602726s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:56.024005466 (596h20m55.895088501s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Uploading piece... Dec 30 11:49:57.704664230 (596h20m57.575747266s) [upload=522750]: Next returned nil segment Dec 30 11:49:57.704753480 (596h20m57.575836535s) [upload=522750]: Waiting for error group managing segments... Dec 30 11:49:58.010161577 (596h20m57.881244612s) [noise=true , address=145.79.227.11:4729 , piece_id=FNSWE3AXBCMJEBSHPTACBCA4LRK4U26URLCRQYCZZGVKCPCCEIZA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010227002 (596h20m57.881310047s) [noise=true , address=145.79.227.10:4403 , piece_id=2CC7DWE7D72CQ7DOAH5VO2LEYJSO2KK2H3KMKIWVXJ26N7R6QQ5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.010858184 (596h20m57.881941229s) [noise=true , address=145.79.227.37:4565 , piece_id=DRQ6KFOLHGG2KUU5UFPSMNRDRSA6BWPY66QGXB6VFYXQEBMI2ISA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.014579990 (596h20m57.885663025s) [noise=true , address=145.79.227.20:4439 , piece_id=H66PJ7VIKYRZUFYAPCT6OJR52R2D4EXTX3URRKBDK32ZOORSI7LQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.017355235 (596h20m57.88843828s) [noise=true , address=145.79.227.39:4015 , piece_id=TA57CUHALASW4RRYQRA5NLSQK3DYIHEFMI2MCTEWTTHQ3ITQUH5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.032446081 (596h20m57.903529116s) [noise=true , address=145.79.227.6:4259 , piece_id=JB5MOJTCGTLHCLL3Q6R4JJYW26SUNFUDHIBBVLIW3WR35FBWGM5A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.056867682 (596h20m57.927950717s) [noise=true , address=145.79.227.50:4375 , piece_id=DBM2TQ2XQV3F47VU5SUG7DFC4FK2ZEKUZEKPMLHLZ223YU3XLIOQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.095062091 (596h20m57.966145136s) [noise=true , address=145.79.227.21:4475 , piece_id=PEWJHMRGT54W73QPHNE4I5TN53XQOVQTT7ZPCVJXYAPTANPLN45A, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.096213222 (596h20m57.967296257s) [noise=true , address=145.79.227.46:4601 , piece_id=WKKJ2XNCDPQY2AEB4AJYH6FH7JVCJAODWEI73I2QJI4ECHUYOTVQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.259666203 (596h20m58.130749238s) [noise=true , address=77.90.31.14:4662 , piece_id=LLWN46HCHDEWWKTH67QR47X24IKMXCJIUOAXVED3TO3CSQJJXWKQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.318930131 (596h20m58.190013166s) [noise=true , address=45.91.116.4:4937 , piece_id=UOSUY6OT2GTIJOO4F5PKHMQ5MVXBHJZ32EZP6IQNVGZQW3DBZTEQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.405343265 (596h20m58.2764263s) [noise=true , address=45.91.116.13:4155 , piece_id=4R6LSVXSJLCQZTSM4SXKTLKPI2R33LTLFVTRVSNZFCDR3X7CZF5Q, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488956378 (596h20m58.360039423s) [noise=true , address=45.91.116.7:4009 , piece_id=XEEDSF63IU2JRJOPGLY3ITMNWUY7ZXIC4AOBBFRJJRURBJ57QQTA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: <nil> Dec 30 11:49:58.488982107 (596h20m58.360065172s) [seg_pos={0 0}, upload=522750]: Segment reached optimal threshold of 13 pieces. Dec 30 11:49:58.489292729 (596h20m58.360375784s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.12:4226: use of closed network connection Dec 30 11:49:58.489302517 (596h20m58.360385562s) [noise=true , address=45.91.116.12:4226 , piece_id=ADJX2ZUYNEQH64FAFY4UNC775HGE7E3S4VJYENP6YK2E7L3EAQSQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489637435 (596h20m58.36072048s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489645791 (596h20m58.360728836s) [noise=true , address=45.91.116.6:4975 , piece_id=BT353VCKPXS6UDXSGD3GWWHFPXUMAWIRT2KTZ6KLOOQD2XUQCLQQ, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489727888 (596h20m58.360810943s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 11:49:58.489750811 (596h20m58.360833876s) [noise=true , address=145.79.227.49:4339 , piece_id=KJNOYCKPLYZIJGTFEDWDKYADTXEWZRVXKJGL4I2OY6UVZNWOHMPA, seg_pos={0 0}, upload=522750]: Long tail cancellation due to optimal threshold Dec 30 11:49:58.489792049 (596h20m58.360875094s) [seg_pos={0 0}, upload=522750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:49:58.490259199 (596h20m58.361342244s) [upload=522750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:49:58.490269318 (596h20m58.361352373s) [upload=522750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:49:58.779262858 (596h20m58.650345913s) [upload=522750]: Done uploading segments... Dec 30 11:49:58.779286784 (596h20m58.650369849s) [upload=522750]: Upload finished. err: <nil> Duration: 30.003786253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..........X...X...........X....X.X........X................] duration: 31.93811298s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:50:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 31.93812705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..........X...X...........X....X.X........X...............] duration: 52.91699797s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:49:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 52.917024391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X...] duration: 4.241886092s trace: https://jaeger.dev.storj.io/trace/6e904e080c2bce20 FAIL (2025-12-30 11:49:49 UTC) orbiter warning: context deadline exceeded Duration: 4.24192722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X.] duration: 5.748341836s trace: https://jaeger.dev.storj.io/trace/56ff714953efb52f FAIL (2025-12-30 11:49:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:48:21.820684079 (596h19m21.691767124s) [upload=522727]: Starting upload Dec 30 11:48:21.820790401 (596h19m21.691873466s) [upload=522727]: Done starting upload Dec 30 11:48:21.820908907 (596h19m21.691991952s) [upload=522727]: Uploading segments... Dec 30 11:48:21.821039355 (596h19m21.6921224s) [upload=522727]: Got next segment. Inline: false Dec 30 11:48:21.821086915 (596h19m21.69216995s) [upload=522727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:48:21.821093338 (596h19m21.692176374s) [upload=522727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:48:22.064127253 (596h19m21.935210319s) [seg_pos={0 0}, upload=522727]: Begin upload segment... Dec 30 11:48:22.064185515 (596h19m21.93526855s) [seg_pos={0 0}, upload=522727]: Done begin upload segment. Dec 30 11:48:22.064217906 (596h19m21.935300982s) [noise=true , address=145.79.227.48:4303 , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.064274914 (596h19m21.935357959s) [noise=true , address=145.79.227.16:4910 , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.065018511 (596h19m21.936101566s) [noise=true , address=77.90.31.11:4555 , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.065678107 (596h19m21.936761142s) [noise=true , address=145.79.227.26:4653 , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.066625571 (596h19m21.937708606s) [noise=true , address=145.79.227.7:4294 , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067091147 (596h19m21.938174182s) [noise=true , address=145.79.227.17:4970 , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067389496 (596h19m21.938472541s) [noise=true , address=145.79.227.47:4266 , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067478334 (596h19m21.938561369s) [noise=true , address=45.91.116.13:4153 , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067859089 (596h19m21.938942134s) [noise=true , address=145.79.227.14:4839 , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068025116 (596h19m21.939108161s) [noise=true , address=145.79.227.5:4221 , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068285190 (596h19m21.939368235s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068674903 (596h19m21.939757948s) [noise=true , address=45.91.116.12:4225 , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068682748 (596h19m21.939765783s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.069094162 (596h19m21.940177197s) [noise=true , address=145.79.227.11:4730 , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.069282400 (596h19m21.940365445s) [noise=true , address=145.79.227.40:4051 , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.070105487 (596h19m21.941188522s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:23.775774806 (596h19m23.646857841s) [upload=522727]: Next returned nil segment Dec 30 11:48:23.775787771 (596h19m23.646870816s) [upload=522727]: Waiting for error group managing segments... Dec 30 11:48:24.070297073 (596h19m23.941380118s) [noise=true , address=145.79.227.5:4221 , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.073745420 (596h19m23.944828455s) [noise=true , address=145.79.227.17:4970 , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.074159528 (596h19m23.945242573s) [noise=true , address=145.79.227.11:4730 , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.074564790 (596h19m23.945647835s) [noise=true , address=145.79.227.7:4294 , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.078996298 (596h19m23.950079343s) [noise=true , address=145.79.227.26:4653 , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.082836511 (596h19m23.953919546s) [noise=true , address=145.79.227.16:4910 , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.135298577 (596h19m24.006381622s) [noise=true , address=145.79.227.14:4839 , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.140443945 (596h19m24.01152699s) [noise=true , address=145.79.227.47:4266 , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.226240736 (596h19m24.097323781s) [noise=true , address=145.79.227.48:4303 , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.483097262 (596h19m24.354180307s) [noise=true , address=45.91.116.13:4153 , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.526902163 (596h19m24.397985219s) [noise=true , address=45.91.116.12:4225 , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.537354927 (596h19m24.408437962s) [noise=true , address=77.90.31.11:4555 , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:25.444901482 (596h19m25.315984527s) [noise=true , address=145.79.227.40:4051 , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:25.444920918 (596h19m25.316003953s) [seg_pos={0 0}, upload=522727]: Segment reached optimal threshold of 13 pieces. Dec 30 11:48:25.445122342 (596h19m25.316205417s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445131970 (596h19m25.316215005s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445296674 (596h19m25.316379709s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445302935 (596h19m25.316385981s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445362118 (596h19m25.316445164s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445372708 (596h19m25.316455753s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445415390 (596h19m25.316498435s) [seg_pos={0 0}, upload=522727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:48:25.445546339 (596h19m25.316629384s) [upload=522727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:48:25.445567920 (596h19m25.316650955s) [upload=522727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:48:25.733181612 (596h19m25.604264657s) [upload=522727]: Done uploading segments... Dec 30 11:48:25.733205869 (596h19m25.604288914s) [upload=522727]: Upload finished. err: <nil> Duration: 5.748383515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X..] duration: 5.002177741s trace: https://jaeger.dev.storj.io/trace/6e904e080c2bce20 FAIL (2025-12-30 11:48:59 UTC) orbiter warning: context deadline exceeded Duration: 5.002204392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..........X...X...........X....X.X........X..............] duration: 48.985987393s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 48.986026689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X....X.......X.....X..XXX....XX.X.X.X........X...X..X..X] duration: 30.003216271s trace: https://jaeger.dev.storj.io/trace/56ff714953efb52f FAIL (2025-12-30 11:48:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:48:21.820684079 (596h19m21.691767124s) [upload=522727]: Starting upload Dec 30 11:48:21.820790401 (596h19m21.691873466s) [upload=522727]: Done starting upload Dec 30 11:48:21.820908907 (596h19m21.691991952s) [upload=522727]: Uploading segments... Dec 30 11:48:21.821039355 (596h19m21.6921224s) [upload=522727]: Got next segment. Inline: false Dec 30 11:48:21.821086915 (596h19m21.69216995s) [upload=522727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:48:21.821093338 (596h19m21.692176374s) [upload=522727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:48:22.064127253 (596h19m21.935210319s) [seg_pos={0 0}, upload=522727]: Begin upload segment... Dec 30 11:48:22.064185515 (596h19m21.93526855s) [seg_pos={0 0}, upload=522727]: Done begin upload segment. Dec 30 11:48:22.064217906 (596h19m21.935300982s) [noise=true , address=145.79.227.48:4303 , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.064274914 (596h19m21.935357959s) [noise=true , address=145.79.227.16:4910 , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.065018511 (596h19m21.936101566s) [noise=true , address=77.90.31.11:4555 , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.065678107 (596h19m21.936761142s) [noise=true , address=145.79.227.26:4653 , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.066625571 (596h19m21.937708606s) [noise=true , address=145.79.227.7:4294 , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067091147 (596h19m21.938174182s) [noise=true , address=145.79.227.17:4970 , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067389496 (596h19m21.938472541s) [noise=true , address=145.79.227.47:4266 , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067478334 (596h19m21.938561369s) [noise=true , address=45.91.116.13:4153 , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.067859089 (596h19m21.938942134s) [noise=true , address=145.79.227.14:4839 , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068025116 (596h19m21.939108161s) [noise=true , address=145.79.227.5:4221 , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068285190 (596h19m21.939368235s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068674903 (596h19m21.939757948s) [noise=true , address=45.91.116.12:4225 , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.068682748 (596h19m21.939765783s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.069094162 (596h19m21.940177197s) [noise=true , address=145.79.227.11:4730 , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.069282400 (596h19m21.940365445s) [noise=true , address=145.79.227.40:4051 , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:22.070105487 (596h19m21.941188522s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Uploading piece... Dec 30 11:48:23.775774806 (596h19m23.646857841s) [upload=522727]: Next returned nil segment Dec 30 11:48:23.775787771 (596h19m23.646870816s) [upload=522727]: Waiting for error group managing segments... Dec 30 11:48:24.070297073 (596h19m23.941380118s) [noise=true , address=145.79.227.5:4221 , piece_id=H3QTJ6HCBM4GPPL5AGME6QCO7EFXUJUSP4FHKIFOAHPMMNDHAOOQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.073745420 (596h19m23.944828455s) [noise=true , address=145.79.227.17:4970 , piece_id=HOHCEKTQ6ZF5DSNYCFI5T5JO3WD6KUBUOUX7ZKOL6Z35VY7K7B5A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.074159528 (596h19m23.945242573s) [noise=true , address=145.79.227.11:4730 , piece_id=YWFBCGFOB5URAFEJ2D7VAGEHDDALV447TOPBNRKYTQHV3USNTLPA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.074564790 (596h19m23.945647835s) [noise=true , address=145.79.227.7:4294 , piece_id=DDYUBNBYEZX75R35QGZVZ6XLEGQHGTIFZP6NFLROD47AGXW2UJ4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.078996298 (596h19m23.950079343s) [noise=true , address=145.79.227.26:4653 , piece_id=RHJAEW5FLOGQW6RY3AOEDTUV3NSZCJTQYZZSGVJCCDSDFJXKNILA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.082836511 (596h19m23.953919546s) [noise=true , address=145.79.227.16:4910 , piece_id=IJGAP45YEWSAW775BNWHP46YV3NHN7U7AQDI3HV7RAQGA47YIGYQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.135298577 (596h19m24.006381622s) [noise=true , address=145.79.227.14:4839 , piece_id=44HX6PWVDEVLXVPPCPIF2HN6JADAB7XMVXSES2MMC3ZVVQ4GJZLQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.140443945 (596h19m24.01152699s) [noise=true , address=145.79.227.47:4266 , piece_id=K36O3Z4TLY3UCTW66L5GMXZ2TVVTG4DUHOGYD3EK34WJZMV2UWSA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.226240736 (596h19m24.097323781s) [noise=true , address=145.79.227.48:4303 , piece_id=Z7FS3RGAFOENNTHYLQY7QQUQV63QAOHTEPPGR4NS4ERG2BSTYCEA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.483097262 (596h19m24.354180307s) [noise=true , address=45.91.116.13:4153 , piece_id=5SR2V5Y4ADFNTTR555SXMW3WITMMECP2LG2D4JMKIMCO767E2Y3Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.526902163 (596h19m24.397985219s) [noise=true , address=45.91.116.12:4225 , piece_id=R6G4EWNBOY2M7NGSWIUBJY7OJIXBENJ2FDBUPSTBMOA6RWH64DJQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:24.537354927 (596h19m24.408437962s) [noise=true , address=77.90.31.11:4555 , piece_id=LKTZSQQUPPJGHPFVEHIBM3JRGKSJZF4UPWLGA55W3TJ32HSPFORA, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:25.444901482 (596h19m25.315984527s) [noise=true , address=145.79.227.40:4051 , piece_id=EM4JHDEOYXVB36O4NANR7DFZZPOEFN54ZOS5FLQLMPXUM4UEVWPQ, seg_pos={0 0}, upload=522727]: Done uploading piece. err: <nil> Dec 30 11:48:25.444920918 (596h19m25.316003953s) [seg_pos={0 0}, upload=522727]: Segment reached optimal threshold of 13 pieces. Dec 30 11:48:25.445122342 (596h19m25.316205417s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445131970 (596h19m25.316215005s) [noise=true , address=145.79.227.33:4871 , piece_id=7ACIV6RHKSES64W5F6OXI5UVOOUSYCUFFDAGPA7X3EZXQQSV5W4A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445296674 (596h19m25.316379709s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445302935 (596h19m25.316385981s) [noise=true , address=77.90.31.15:4700 , piece_id=JQIQBLFZFM7UBKKE3RRPXJVVHLKCMQNAQUU757XY2VPHZHGFUG6Q, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445362118 (596h19m25.316445164s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 11:48:25.445372708 (596h19m25.316455753s) [noise=true , address=145.79.227.49:4338 , piece_id=742JLRH5YF5BKUWI7YPOYPKTWBIAPQ5NQMIIRGE4B4AVXC3J7J7A, seg_pos={0 0}, upload=522727]: Long tail cancellation due to optimal threshold Dec 30 11:48:25.445415390 (596h19m25.316498435s) [seg_pos={0 0}, upload=522727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:48:25.445546339 (596h19m25.316629384s) [upload=522727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:48:25.445567920 (596h19m25.316650955s) [upload=522727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:48:25.733181612 (596h19m25.604264657s) [upload=522727]: Done uploading segments... Dec 30 11:48:25.733205869 (596h19m25.604288914s) [upload=522727]: Upload finished. err: <nil> Duration: 30.003319597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X....X.......X.....X..XXX....XX.X.X.X........X...X..X..] duration: 5.875487851s trace: https://jaeger.dev.storj.io/trace/37bbc15678dcb08d FAIL (2025-12-30 11:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ) context deadline exceeded ======LOGS====== Dec 30 11:46:16.208132621 (596h17m16.079215657s) [upload=522695]: Starting upload Dec 30 11:46:16.208186665 (596h17m16.07926971s) [upload=522695]: Done starting upload Dec 30 11:46:16.208323715 (596h17m16.07940675s) [upload=522695]: Uploading segments... Dec 30 11:46:16.208437532 (596h17m16.079520567s) [upload=522695]: Got next segment. Inline: false Dec 30 11:46:16.208509760 (596h17m16.079592795s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:46:16.208515220 (596h17m16.079598255s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:46:16.454687231 (596h17m16.325770266s) [seg_pos={0 0}, upload=522695]: Begin upload segment... Dec 30 11:46:16.454753057 (596h17m16.325836092s) [seg_pos={0 0}, upload=522695]: Done begin upload segment. Dec 30 11:46:16.454796579 (596h17m16.325879614s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.454909324 (596h17m16.325992369s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.455692065 (596h17m16.32677516s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456031170 (596h17m16.327114215s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456265256 (596h17m16.327348301s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456370787 (596h17m16.327453822s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456642865 (596h17m16.3277259s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456819411 (596h17m16.327902446s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457042716 (596h17m16.328125751s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457341996 (596h17m16.328425031s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457495288 (596h17m16.328578333s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457613854 (596h17m16.328696899s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458139254 (596h17m16.329222299s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458155886 (596h17m16.329238931s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458789002 (596h17m16.329872047s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.459435062 (596h17m16.330518107s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:18.064524931 (596h17m17.935607966s) [upload=522695]: Next returned nil segment Dec 30 11:46:18.064543617 (596h17m17.935626662s) [upload=522695]: Waiting for error group managing segments... Dec 30 11:46:18.445650235 (596h17m18.31673327s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.445674992 (596h17m18.316758037s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.446391677 (596h17m18.317474712s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.447931178 (596h17m18.319014224s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.450052899 (596h17m18.321135944s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.458643858 (596h17m18.329726903s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.462219195 (596h17m18.33330224s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.463608330 (596h17m18.334691365s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.473171460 (596h17m18.344254495s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.474467548 (596h17m18.345550583s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.732587350 (596h17m18.603670395s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.746855699 (596h17m18.617938744s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975180716 (596h17m18.846263752s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975208249 (596h17m18.846291294s) [seg_pos={0 0}, upload=522695]: Segment reached optimal threshold of 13 pieces. Dec 30 11:46:18.975470548 (596h17m18.846553613s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975480316 (596h17m18.846563361s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975756262 (596h17m18.846839307s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:43450->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:46:18.975782592 (596h17m18.846865657s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975843028 (596h17m18.846926083s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975855421 (596h17m18.846938466s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975883404 (596h17m18.846966439s) [seg_pos={0 0}, upload=522695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:46:18.975997271 (596h17m18.847080326s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:46:18.976005808 (596h17m18.847088853s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:46:19.262987707 (596h17m19.134070752s) [upload=522695]: Done uploading segments... Dec 30 11:46:19.263009128 (596h17m19.134092163s) [upload=522695]: Upload finished. err: <nil> Duration: 5.875518168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X.] duration: 3.462359518s trace: https://jaeger.dev.storj.io/trace/6e904e080c2bce20 FAIL (2025-12-30 11:47:47 UTC) orbiter warning: context deadline exceeded Duration: 3.462476801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..........X...X...........X....X.X........X.............] duration: 51.759723491s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:47:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 51.759752186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...X] duration: 5.001559201s trace: https://jaeger.dev.storj.io/trace/6e904e080c2bce20 FAIL (2025-12-30 11:47:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001581864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X...] duration: 5.001583737s trace: https://jaeger.dev.storj.io/trace/17445fb928c372b5 FAIL (2025-12-30 11:47:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001660183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X....X.......X.....X..XXX....XX.X.X.X........X...X..X.] duration: 6.010534724s trace: https://jaeger.dev.storj.io/trace/37bbc15678dcb08d FAIL (2025-12-30 11:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ) context deadline exceeded ======LOGS====== Dec 30 11:46:16.208132621 (596h17m16.079215657s) [upload=522695]: Starting upload Dec 30 11:46:16.208186665 (596h17m16.07926971s) [upload=522695]: Done starting upload Dec 30 11:46:16.208323715 (596h17m16.07940675s) [upload=522695]: Uploading segments... Dec 30 11:46:16.208437532 (596h17m16.079520567s) [upload=522695]: Got next segment. Inline: false Dec 30 11:46:16.208509760 (596h17m16.079592795s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:46:16.208515220 (596h17m16.079598255s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:46:16.454687231 (596h17m16.325770266s) [seg_pos={0 0}, upload=522695]: Begin upload segment... Dec 30 11:46:16.454753057 (596h17m16.325836092s) [seg_pos={0 0}, upload=522695]: Done begin upload segment. Dec 30 11:46:16.454796579 (596h17m16.325879614s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.454909324 (596h17m16.325992369s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.455692065 (596h17m16.32677516s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456031170 (596h17m16.327114215s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456265256 (596h17m16.327348301s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456370787 (596h17m16.327453822s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456642865 (596h17m16.3277259s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456819411 (596h17m16.327902446s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457042716 (596h17m16.328125751s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457341996 (596h17m16.328425031s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457495288 (596h17m16.328578333s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457613854 (596h17m16.328696899s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458139254 (596h17m16.329222299s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458155886 (596h17m16.329238931s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458789002 (596h17m16.329872047s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.459435062 (596h17m16.330518107s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:18.064524931 (596h17m17.935607966s) [upload=522695]: Next returned nil segment Dec 30 11:46:18.064543617 (596h17m17.935626662s) [upload=522695]: Waiting for error group managing segments... Dec 30 11:46:18.445650235 (596h17m18.31673327s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.445674992 (596h17m18.316758037s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.446391677 (596h17m18.317474712s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.447931178 (596h17m18.319014224s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.450052899 (596h17m18.321135944s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.458643858 (596h17m18.329726903s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.462219195 (596h17m18.33330224s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.463608330 (596h17m18.334691365s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.473171460 (596h17m18.344254495s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.474467548 (596h17m18.345550583s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.732587350 (596h17m18.603670395s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.746855699 (596h17m18.617938744s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975180716 (596h17m18.846263752s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975208249 (596h17m18.846291294s) [seg_pos={0 0}, upload=522695]: Segment reached optimal threshold of 13 pieces. Dec 30 11:46:18.975470548 (596h17m18.846553613s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975480316 (596h17m18.846563361s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975756262 (596h17m18.846839307s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:43450->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:46:18.975782592 (596h17m18.846865657s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975843028 (596h17m18.846926083s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975855421 (596h17m18.846938466s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975883404 (596h17m18.846966439s) [seg_pos={0 0}, upload=522695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:46:18.975997271 (596h17m18.847080326s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:46:18.976005808 (596h17m18.847088853s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:46:19.262987707 (596h17m19.134070752s) [upload=522695]: Done uploading segments... Dec 30 11:46:19.263009128 (596h17m19.134092163s) [upload=522695]: Upload finished. err: <nil> Duration: 6.010562827s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X..] duration: 3.677345557s trace: https://jaeger.dev.storj.io/trace/17445fb928c372b5 FAIL (2025-12-30 11:46:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.677368791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X....X.......X.....X..XXX....XX.X.X.X........X...X..X] duration: 30.002563442s trace: https://jaeger.dev.storj.io/trace/37bbc15678dcb08d FAIL (2025-12-30 11:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ) context deadline exceeded ======LOGS====== Dec 30 11:46:16.208132621 (596h17m16.079215657s) [upload=522695]: Starting upload Dec 30 11:46:16.208186665 (596h17m16.07926971s) [upload=522695]: Done starting upload Dec 30 11:46:16.208323715 (596h17m16.07940675s) [upload=522695]: Uploading segments... Dec 30 11:46:16.208437532 (596h17m16.079520567s) [upload=522695]: Got next segment. Inline: false Dec 30 11:46:16.208509760 (596h17m16.079592795s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:46:16.208515220 (596h17m16.079598255s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:46:16.454687231 (596h17m16.325770266s) [seg_pos={0 0}, upload=522695]: Begin upload segment... Dec 30 11:46:16.454753057 (596h17m16.325836092s) [seg_pos={0 0}, upload=522695]: Done begin upload segment. Dec 30 11:46:16.454796579 (596h17m16.325879614s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.454909324 (596h17m16.325992369s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.455692065 (596h17m16.32677516s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456031170 (596h17m16.327114215s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456265256 (596h17m16.327348301s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456370787 (596h17m16.327453822s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456642865 (596h17m16.3277259s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.456819411 (596h17m16.327902446s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457042716 (596h17m16.328125751s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457341996 (596h17m16.328425031s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457495288 (596h17m16.328578333s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.457613854 (596h17m16.328696899s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458139254 (596h17m16.329222299s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458155886 (596h17m16.329238931s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.458789002 (596h17m16.329872047s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:16.459435062 (596h17m16.330518107s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Uploading piece... Dec 30 11:46:18.064524931 (596h17m17.935607966s) [upload=522695]: Next returned nil segment Dec 30 11:46:18.064543617 (596h17m17.935626662s) [upload=522695]: Waiting for error group managing segments... Dec 30 11:46:18.445650235 (596h17m18.31673327s) [noise=true , address=145.79.227.21:4473 , piece_id=WXWZT2DBVZICUQJKIMPWV3S5EJ3RCYZTED3HKV2IWWUKHZASXATQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.445674992 (596h17m18.316758037s) [noise=true , address=145.79.227.51:4410 , piece_id=GYNFLKVJJFXYH5U7GEZPQOENDV5CUUXKZC6TPP6BID442XREULXQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.446391677 (596h17m18.317474712s) [noise=true , address=145.79.227.16:4912 , piece_id=A6RW7GWAO3CQ23KB6YP3KBPRLCTOXPJ4JHHLGLWLKBQEYUNVU4DQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.447931178 (596h17m18.319014224s) [noise=true , address=145.79.227.32:4835 , piece_id=SRMO7MY7EDVOUNXZ32VRYYDAAMSVAQW7XO7CN4R4VXF2CF6PEUBQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.450052899 (596h17m18.321135944s) [noise=true , address=145.79.227.20:4439 , piece_id=C26P4C2GVNOTAQP2KWCV33X7OYTBMZF2EDROUTKV2AX5RUEKFEDA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.458643858 (596h17m18.329726903s) [noise=true , address=145.79.227.11:4730 , piece_id=ZD45W436TV3ABZSQK5PHHXQOFNCY3F3467GKLKSLH5ZJ2VINF3PQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.462219195 (596h17m18.33330224s) [noise=true , address=145.79.227.5:4221 , piece_id=XG7IJU34LJF6LYGX5IGZ7GVIJ6CWLBX4QNBNYIGB5G74OG2EB46A, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.463608330 (596h17m18.334691365s) [noise=true , address=145.79.227.36:4517 , piece_id=XRVJLZEMU5F23X3NQLR7OTIP3TN6PHXTDA2FAVR6GTPWAM6LNMTA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.473171460 (596h17m18.344254495s) [noise=true , address=145.79.227.23:4547 , piece_id=CNDA3CJDDJABCRYAAMDKGSKPMHSZATXNSA4HGUD7QQZR7KXACC5Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.474467548 (596h17m18.345550583s) [noise=true , address=145.79.227.47:4267 , piece_id=DEMDH5WGCAY4VVNVFELFZYKMLXK4DP2YCLJEMVVEQBH46TB3FMDQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.732587350 (596h17m18.603670395s) [noise=true , address=77.90.31.18:4805 , piece_id=ZB6IPEE2JYXVV3BZQ752MS6CU6GRP575J6JPSJI5YP2TUW6W34RQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.746855699 (596h17m18.617938744s) [noise=true , address=77.90.31.19:4842 , piece_id=74IVPALWDT2FXDYWITUKBH6ESUQDWVPND4G6ZOEUTJLFLL2ESG7Q, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975180716 (596h17m18.846263752s) [noise=true , address=45.91.116.4:4940 , piece_id=556HJJKVPIZSOVGTR3UTABDUFFIBR6RKZ6WWTQ2GAX4BL7GHAEUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: <nil> Dec 30 11:46:18.975208249 (596h17m18.846291294s) [seg_pos={0 0}, upload=522695]: Segment reached optimal threshold of 13 pieces. Dec 30 11:46:18.975470548 (596h17m18.846553613s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975480316 (596h17m18.846563361s) [noise=true , address=45.91.116.8:4047 , piece_id=ZQQWI2OEHNXALSK7HPUY6KOQ4VWX45X3T27OHNVLTWYL7BMMARUA, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975756262 (596h17m18.846839307s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:43450->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 11:46:18.975782592 (596h17m18.846865657s) [noise=true , address=45.91.116.6:4976 , piece_id=AC4HY6R7JIDEXHR44BYWSER5DXOF2NXQL5W5K5K3CN6W2YFEW4ZQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975843028 (596h17m18.846926083s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:46:18.975855421 (596h17m18.846938466s) [noise=true , address=45.91.116.10:4084 , piece_id=XPABSFLFDV52NJH7BCBAVQCTB3RWOVQM7P7XNYVXHJREBICVQMRQ, seg_pos={0 0}, upload=522695]: Long tail cancellation due to optimal threshold Dec 30 11:46:18.975883404 (596h17m18.846966439s) [seg_pos={0 0}, upload=522695]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:46:18.975997271 (596h17m18.847080326s) [upload=522695]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:46:18.976005808 (596h17m18.847088853s) [upload=522695]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:46:19.262987707 (596h17m19.134070752s) [upload=522695]: Done uploading segments... Dec 30 11:46:19.263009128 (596h17m19.134092163s) [upload=522695]: Upload finished. err: <nil> Duration: 30.002587257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..........X...X...........X....X.X........X............] duration: 47.224329487s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 47.2243622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X.] duration: 3.427382798s trace: https://jaeger.dev.storj.io/trace/17445fb928c372b5 FAIL (2025-12-30 11:46:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42741057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........X....X.......X.....X..XXX....XX.X.X.X........X...X..] duration: 5.963402175s trace: https://jaeger.dev.storj.io/trace/52643a39ff7bea40 FAIL (2025-12-30 11:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:43:41.838676675 (596h14m41.70975972s) [upload=522658]: Starting upload Dec 30 11:43:41.838752799 (596h14m41.709835835s) [upload=522658]: Done starting upload Dec 30 11:43:41.838978990 (596h14m41.710062025s) [upload=522658]: Uploading segments... Dec 30 11:43:41.839025339 (596h14m41.710108384s) [upload=522658]: Got next segment. Inline: false Dec 30 11:43:41.839071717 (596h14m41.710154762s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:43:41.839078259 (596h14m41.710161304s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:43:42.086773771 (596h14m41.957856806s) [seg_pos={0 0}, upload=522658]: Begin upload segment... Dec 30 11:43:42.086818396 (596h14m41.957901431s) [seg_pos={0 0}, upload=522658]: Done begin upload segment. Dec 30 11:43:42.086844015 (596h14m41.95792705s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.090717060 (596h14m41.961800095s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.091820431 (596h14m41.962903476s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.093009415 (596h14m41.96409245s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.094080064 (596h14m41.965163099s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.095328531 (596h14m41.966411566s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.096663122 (596h14m41.967746157s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.097886261 (596h14m41.968969296s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.099429259 (596h14m41.970512294s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.101018646 (596h14m41.972101691s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.102423531 (596h14m41.973506576s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.104064615 (596h14m41.975147651s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.105423583 (596h14m41.976506628s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.107818684 (596h14m41.978901719s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.111183291 (596h14m41.982266336s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.117665274 (596h14m41.988748319s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:43.725446618 (596h14m43.596529653s) [upload=522658]: Next returned nil segment Dec 30 11:43:43.725463920 (596h14m43.596546975s) [upload=522658]: Waiting for error group managing segments... Dec 30 11:43:44.106208669 (596h14m43.977291714s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.107281202 (596h14m43.978364237s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.110341399 (596h14m43.981424434s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.120936935 (596h14m43.99201997s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.126103222 (596h14m43.997186258s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.130375167 (596h14m44.001458202s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.133276712 (596h14m44.004359747s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.144565117 (596h14m44.015648162s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.198303183 (596h14m44.069386228s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.353984124 (596h14m44.225067159s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.485061857 (596h14m44.356144892s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.568079045 (596h14m44.43916208s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694795112 (596h14m44.565878147s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694811624 (596h14m44.565894669s) [seg_pos={0 0}, upload=522658]: Segment reached optimal threshold of 13 pieces. Dec 30 11:43:44.695021885 (596h14m44.56610492s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection Dec 30 11:43:44.695030431 (596h14m44.566113466s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695408140 (596h14m44.566491185s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695417588 (596h14m44.566500623s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695543418 (596h14m44.566626463s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695551112 (596h14m44.566634147s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695570209 (596h14m44.566653244s) [seg_pos={0 0}, upload=522658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:43:44.695657905 (596h14m44.566740951s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:43:44.695664047 (596h14m44.566747092s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:43:45.015743450 (596h14m44.886826505s) [upload=522658]: Done uploading segments... Dec 30 11:43:45.015772816 (596h14m44.886855851s) [upload=522658]: Upload finished. err: <nil> Duration: 5.963430899s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.X] duration: 5.001863677s trace: https://jaeger.dev.storj.io/trace/17445fb928c372b5 FAIL (2025-12-30 11:45:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001885029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..........X...X...........X....X.X........X...........] duration: 15.510517679s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:44:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 15.51053425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........X....X.......X.....X..XXX....XX.X.X.X........X...X.] duration: 5.792314867s trace: https://jaeger.dev.storj.io/trace/52643a39ff7bea40 FAIL (2025-12-30 11:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:43:41.838676675 (596h14m41.70975972s) [upload=522658]: Starting upload Dec 30 11:43:41.838752799 (596h14m41.709835835s) [upload=522658]: Done starting upload Dec 30 11:43:41.838978990 (596h14m41.710062025s) [upload=522658]: Uploading segments... Dec 30 11:43:41.839025339 (596h14m41.710108384s) [upload=522658]: Got next segment. Inline: false Dec 30 11:43:41.839071717 (596h14m41.710154762s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:43:41.839078259 (596h14m41.710161304s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:43:42.086773771 (596h14m41.957856806s) [seg_pos={0 0}, upload=522658]: Begin upload segment... Dec 30 11:43:42.086818396 (596h14m41.957901431s) [seg_pos={0 0}, upload=522658]: Done begin upload segment. Dec 30 11:43:42.086844015 (596h14m41.95792705s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.090717060 (596h14m41.961800095s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.091820431 (596h14m41.962903476s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.093009415 (596h14m41.96409245s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.094080064 (596h14m41.965163099s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.095328531 (596h14m41.966411566s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.096663122 (596h14m41.967746157s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.097886261 (596h14m41.968969296s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.099429259 (596h14m41.970512294s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.101018646 (596h14m41.972101691s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.102423531 (596h14m41.973506576s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.104064615 (596h14m41.975147651s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.105423583 (596h14m41.976506628s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.107818684 (596h14m41.978901719s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.111183291 (596h14m41.982266336s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.117665274 (596h14m41.988748319s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:43.725446618 (596h14m43.596529653s) [upload=522658]: Next returned nil segment Dec 30 11:43:43.725463920 (596h14m43.596546975s) [upload=522658]: Waiting for error group managing segments... Dec 30 11:43:44.106208669 (596h14m43.977291714s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.107281202 (596h14m43.978364237s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.110341399 (596h14m43.981424434s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.120936935 (596h14m43.99201997s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.126103222 (596h14m43.997186258s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.130375167 (596h14m44.001458202s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.133276712 (596h14m44.004359747s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.144565117 (596h14m44.015648162s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.198303183 (596h14m44.069386228s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.353984124 (596h14m44.225067159s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.485061857 (596h14m44.356144892s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.568079045 (596h14m44.43916208s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694795112 (596h14m44.565878147s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694811624 (596h14m44.565894669s) [seg_pos={0 0}, upload=522658]: Segment reached optimal threshold of 13 pieces. Dec 30 11:43:44.695021885 (596h14m44.56610492s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection Dec 30 11:43:44.695030431 (596h14m44.566113466s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695408140 (596h14m44.566491185s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695417588 (596h14m44.566500623s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695543418 (596h14m44.566626463s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695551112 (596h14m44.566634147s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695570209 (596h14m44.566653244s) [seg_pos={0 0}, upload=522658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:43:44.695657905 (596h14m44.566740951s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:43:44.695664047 (596h14m44.566747092s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:43:45.015743450 (596h14m44.886826505s) [upload=522658]: Done uploading segments... Dec 30 11:43:45.015772816 (596h14m44.886855851s) [upload=522658]: Upload finished. err: <nil> Duration: 5.792333893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..........X...X...........X....X.X........X..........] duration: 31.56159898s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 31.561617426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X.] duration: 4.774806709s trace: https://jaeger.dev.storj.io/trace/11bdea12a14d7201 FAIL (2025-12-30 11:44:37 UTC) orbiter warning: context deadline exceeded Duration: 4.774831647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........X....X.......X.....X..XXX....XX.X.X.X........X...X] duration: 30.00401864s trace: https://jaeger.dev.storj.io/trace/52643a39ff7bea40 FAIL (2025-12-30 11:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 11:43:41.838676675 (596h14m41.70975972s) [upload=522658]: Starting upload Dec 30 11:43:41.838752799 (596h14m41.709835835s) [upload=522658]: Done starting upload Dec 30 11:43:41.838978990 (596h14m41.710062025s) [upload=522658]: Uploading segments... Dec 30 11:43:41.839025339 (596h14m41.710108384s) [upload=522658]: Got next segment. Inline: false Dec 30 11:43:41.839071717 (596h14m41.710154762s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:43:41.839078259 (596h14m41.710161304s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:43:42.086773771 (596h14m41.957856806s) [seg_pos={0 0}, upload=522658]: Begin upload segment... Dec 30 11:43:42.086818396 (596h14m41.957901431s) [seg_pos={0 0}, upload=522658]: Done begin upload segment. Dec 30 11:43:42.086844015 (596h14m41.95792705s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.090717060 (596h14m41.961800095s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.091820431 (596h14m41.962903476s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.093009415 (596h14m41.96409245s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.094080064 (596h14m41.965163099s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.095328531 (596h14m41.966411566s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.096663122 (596h14m41.967746157s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.097886261 (596h14m41.968969296s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.099429259 (596h14m41.970512294s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.101018646 (596h14m41.972101691s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.102423531 (596h14m41.973506576s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.104064615 (596h14m41.975147651s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.105423583 (596h14m41.976506628s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.107818684 (596h14m41.978901719s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.111183291 (596h14m41.982266336s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:42.117665274 (596h14m41.988748319s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Uploading piece... Dec 30 11:43:43.725446618 (596h14m43.596529653s) [upload=522658]: Next returned nil segment Dec 30 11:43:43.725463920 (596h14m43.596546975s) [upload=522658]: Waiting for error group managing segments... Dec 30 11:43:44.106208669 (596h14m43.977291714s) [noise=true , address=145.79.227.3:4150 , piece_id=N2NSQOOTXV2F4LZPOQ2JYLK5KSM3RCSRJZFLREYF4LCOJLBLGITA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.107281202 (596h14m43.978364237s) [noise=true , address=145.79.227.26:4654 , piece_id=LFDKTGDNEYCCFOC53BRT6V4JSXCSJJATPZHVTMWFARRZVJUBFMEQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.110341399 (596h14m43.981424434s) [noise=true , address=145.79.227.17:4970 , piece_id=QPRAWQ6I4APDSCMEVLAZTEBX3UUYI3WHRM6T4GZZVW2R7LRA3AYQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.120936935 (596h14m43.99201997s) [noise=true , address=145.79.227.11:4729 , piece_id=H23BZKEAAIUCRCJJ44QID3I4KHFMEP7OWHI7A5XKZJQG4Q5XUKMQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.126103222 (596h14m43.997186258s) [noise=true , address=145.79.227.15:4876 , piece_id=QLZUTCW7TO7EDO5I6LHI65KNILARIJPZNDOUZ3O25K72SK37J5RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.130375167 (596h14m44.001458202s) [noise=true , address=145.79.227.41:4086 , piece_id=MTVAFALBYNJRDP2HJVF4CTMBIA4DIHPSSKDVP6N4VC7FG4WMT4MQ, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.133276712 (596h14m44.004359747s) [noise=true , address=145.79.227.7:4295 , piece_id=XLGMI4EZ7WFFI5NIWTKZR6E2PXHHCVMLHFLE5VQSLB7QBMORTCKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.144565117 (596h14m44.015648162s) [noise=true , address=145.79.227.47:4267 , piece_id=IB7DBH5OG5DEYYPPSW3PW4KSCVQ4SBRLFFCVFHTF23DUWTHMMN3A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.198303183 (596h14m44.069386228s) [noise=true , address=145.79.227.32:4834 , piece_id=KCSLAPOHJPXAUYL4GNHSCVCKEIZ5KYVAXQEDDRD6HLYWALW5ZLKA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.353984124 (596h14m44.225067159s) [noise=true , address=77.90.31.15:4700 , piece_id=E6WKUIDTONM4HT6KLDQCU6J2DVF77TJHWEU7E6VZVW7Y3QDHH3RA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.485061857 (596h14m44.356144892s) [noise=true , address=45.91.116.6:4974 , piece_id=B432NHKFRN55YXSCJN534HE6TG3NCNUZX4E36GP7AUE7EQDUMMAA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.568079045 (596h14m44.43916208s) [noise=true , address=45.91.116.14:4190 , piece_id=PYYR5RZGWSDCEMWOURYXQMDB5PTYBAAO3R3PRBX3BAF65WO4KF6Q, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694795112 (596h14m44.565878147s) [noise=true , address=77.90.31.14:4661 , piece_id=2P64UTYUIJRJNHPOSYTXWTW3D3E54QOSNMBFZYVXOL7BGARPWRHA, seg_pos={0 0}, upload=522658]: Done uploading piece. err: <nil> Dec 30 11:43:44.694811624 (596h14m44.565894669s) [seg_pos={0 0}, upload=522658]: Segment reached optimal threshold of 13 pieces. Dec 30 11:43:44.695021885 (596h14m44.56610492s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45618->77.90.31.13:4627: use of closed network connection Dec 30 11:43:44.695030431 (596h14m44.566113466s) [noise=true , address=77.90.31.13:4627 , piece_id=DVATGQFNNMKIDAXDAF344Q5U7BVOX442JHMW3QY3PMI26VM34A2A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695408140 (596h14m44.566491185s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695417588 (596h14m44.566500623s) [noise=true , address=145.79.227.40:4050 , piece_id=ZM7WHUQWSVBIRIHNZXE3NKEDMQSJ62LQF3AX4EOG6KCWVP5VIN4A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695543418 (596h14m44.566626463s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 11:43:44.695551112 (596h14m44.566634147s) [noise=true , address=77.90.31.21:4880 , piece_id=R725LBDXDQBDVOE54KDVY5FVINTER3R7O4ETPN6LPB6YTYYLM75A, seg_pos={0 0}, upload=522658]: Long tail cancellation due to optimal threshold Dec 30 11:43:44.695570209 (596h14m44.566653244s) [seg_pos={0 0}, upload=522658]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:43:44.695657905 (596h14m44.566740951s) [upload=522658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:43:44.695664047 (596h14m44.566747092s) [upload=522658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:43:45.015743450 (596h14m44.886826505s) [upload=522658]: Done uploading segments... Dec 30 11:43:45.015772816 (596h14m44.886855851s) [upload=522658]: Upload finished. err: <nil> Duration: 30.00403914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..........X...X...........X....X.X........X.........] duration: 35.911053299s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 35.911078718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..........X...X...........X....X.X........X........] duration: 54.318694958s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:43:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 54.318711169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.X] duration: 5.00072395s trace: https://jaeger.dev.storj.io/trace/11bdea12a14d7201 FAIL (2025-12-30 11:43:25 UTC) orbiter warning: context deadline exceeded Duration: 5.000777492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X.] duration: 5.000981739s trace: https://jaeger.dev.storj.io/trace/678a80fe505faafa FAIL (2025-12-30 11:43:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001005856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..........X....X.......X.....X..XXX....XX.X.X.X........X...] duration: 5.688594908s trace: https://jaeger.dev.storj.io/trace/5675c8f3d22b61db FAIL (2025-12-30 11:42:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ) context deadline exceeded ======LOGS====== Dec 30 11:39:33.857167265 (596h10m33.7282503s) [upload=522603]: Starting upload Dec 30 11:39:33.857244021 (596h10m33.728327086s) [upload=522603]: Done starting upload Dec 30 11:39:33.857390640 (596h10m33.728473675s) [upload=522603]: Uploading segments... Dec 30 11:39:33.857551617 (596h10m33.728634652s) [upload=522603]: Got next segment. Inline: false Dec 30 11:39:33.857584299 (596h10m33.728667334s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:39:33.857589389 (596h10m33.728672424s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:39:34.102965955 (596h10m33.97404899s) [seg_pos={0 0}, upload=522603]: Begin upload segment... Dec 30 11:39:34.103006453 (596h10m33.974089488s) [seg_pos={0 0}, upload=522603]: Done begin upload segment. Dec 30 11:39:34.103029777 (596h10m33.974112812s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103195282 (596h10m33.974278327s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103667051 (596h10m33.974750086s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103912248 (596h10m33.974995283s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104116967 (596h10m33.975200032s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104268255 (596h10m33.975351291s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104427067 (596h10m33.975510112s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104556694 (596h10m33.975639729s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104766183 (596h10m33.975849218s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104863469 (596h10m33.975946504s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105084309 (596h10m33.976167344s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105175323 (596h10m33.976258368s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105500431 (596h10m33.976583466s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105759916 (596h10m33.976842951s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106158264 (596h10m33.977241299s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106444299 (596h10m33.977527334s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:35.810341633 (596h10m35.681424668s) [upload=522603]: Next returned nil segment Dec 30 11:39:35.810481680 (596h10m35.681564726s) [upload=522603]: Waiting for error group managing segments... Dec 30 11:39:36.089501399 (596h10m35.960584434s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.092835126 (596h10m35.963918161s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.097761046 (596h10m35.968844081s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.102058158 (596h10m35.973141193s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.107038902 (596h10m35.978121938s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.114852431 (596h10m35.985935466s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.119241207 (596h10m35.990324242s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121297313 (596h10m35.992380348s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121677617 (596h10m35.992760652s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.178766824 (596h10m36.049849869s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.291087951 (596h10m36.162170997s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.296504245 (596h10m36.167587281s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301077393 (596h10m36.172160428s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301098233 (596h10m36.172181268s) [seg_pos={0 0}, upload=522603]: Segment reached optimal threshold of 13 pieces. Dec 30 11:39:36.301289016 (596h10m36.172372061s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301301980 (596h10m36.172385045s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301453720 (596h10m36.172536765s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301490470 (596h10m36.172573505s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301648460 (596h10m36.172731495s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301656837 (596h10m36.172739872s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301671514 (596h10m36.172754559s) [seg_pos={0 0}, upload=522603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:39:36.301740436 (596h10m36.172823471s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:39:36.301745044 (596h10m36.172828079s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:39:36.591390145 (596h10m36.46247319s) [upload=522603]: Done uploading segments... Dec 30 11:39:36.591416415 (596h10m36.46249946s) [upload=522603]: Upload finished. err: <nil> Duration: 5.688618202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..........X...X...........X....X.X........X.......] duration: 58.886566991s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:42:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 58.886600735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..X] duration: 5.001519587s trace: https://jaeger.dev.storj.io/trace/678a80fe505faafa FAIL (2025-12-30 11:42:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001534325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..........X....X.......X.....X..XXX....XX.X.X.X........X..] duration: 5.794343378s trace: https://jaeger.dev.storj.io/trace/5675c8f3d22b61db FAIL (2025-12-30 11:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ) context deadline exceeded ======LOGS====== Dec 30 11:39:33.857167265 (596h10m33.7282503s) [upload=522603]: Starting upload Dec 30 11:39:33.857244021 (596h10m33.728327086s) [upload=522603]: Done starting upload Dec 30 11:39:33.857390640 (596h10m33.728473675s) [upload=522603]: Uploading segments... Dec 30 11:39:33.857551617 (596h10m33.728634652s) [upload=522603]: Got next segment. Inline: false Dec 30 11:39:33.857584299 (596h10m33.728667334s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:39:33.857589389 (596h10m33.728672424s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:39:34.102965955 (596h10m33.97404899s) [seg_pos={0 0}, upload=522603]: Begin upload segment... Dec 30 11:39:34.103006453 (596h10m33.974089488s) [seg_pos={0 0}, upload=522603]: Done begin upload segment. Dec 30 11:39:34.103029777 (596h10m33.974112812s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103195282 (596h10m33.974278327s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103667051 (596h10m33.974750086s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103912248 (596h10m33.974995283s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104116967 (596h10m33.975200032s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104268255 (596h10m33.975351291s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104427067 (596h10m33.975510112s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104556694 (596h10m33.975639729s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104766183 (596h10m33.975849218s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104863469 (596h10m33.975946504s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105084309 (596h10m33.976167344s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105175323 (596h10m33.976258368s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105500431 (596h10m33.976583466s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105759916 (596h10m33.976842951s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106158264 (596h10m33.977241299s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106444299 (596h10m33.977527334s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:35.810341633 (596h10m35.681424668s) [upload=522603]: Next returned nil segment Dec 30 11:39:35.810481680 (596h10m35.681564726s) [upload=522603]: Waiting for error group managing segments... Dec 30 11:39:36.089501399 (596h10m35.960584434s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.092835126 (596h10m35.963918161s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.097761046 (596h10m35.968844081s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.102058158 (596h10m35.973141193s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.107038902 (596h10m35.978121938s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.114852431 (596h10m35.985935466s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.119241207 (596h10m35.990324242s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121297313 (596h10m35.992380348s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121677617 (596h10m35.992760652s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.178766824 (596h10m36.049849869s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.291087951 (596h10m36.162170997s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.296504245 (596h10m36.167587281s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301077393 (596h10m36.172160428s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301098233 (596h10m36.172181268s) [seg_pos={0 0}, upload=522603]: Segment reached optimal threshold of 13 pieces. Dec 30 11:39:36.301289016 (596h10m36.172372061s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301301980 (596h10m36.172385045s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301453720 (596h10m36.172536765s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301490470 (596h10m36.172573505s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301648460 (596h10m36.172731495s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301656837 (596h10m36.172739872s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301671514 (596h10m36.172754559s) [seg_pos={0 0}, upload=522603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:39:36.301740436 (596h10m36.172823471s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:39:36.301745044 (596h10m36.172828079s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:39:36.591390145 (596h10m36.46247319s) [upload=522603]: Done uploading segments... Dec 30 11:39:36.591416415 (596h10m36.46249946s) [upload=522603]: Upload finished. err: <nil> Duration: 5.794366773s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X..] duration: 4.038476746s trace: https://jaeger.dev.storj.io/trace/3aa40edc5615884b FAIL (2025-12-30 11:41:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.038504609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X..........X...X...........X....X.X........X......] duration: 15.451235065s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 15.451249724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X..........X....X.......X.....X..XXX....XX.X.X.X........X.] duration: 5.786149813s trace: https://jaeger.dev.storj.io/trace/5675c8f3d22b61db FAIL (2025-12-30 11:40:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ) context deadline exceeded ======LOGS====== Dec 30 11:39:33.857167265 (596h10m33.7282503s) [upload=522603]: Starting upload Dec 30 11:39:33.857244021 (596h10m33.728327086s) [upload=522603]: Done starting upload Dec 30 11:39:33.857390640 (596h10m33.728473675s) [upload=522603]: Uploading segments... Dec 30 11:39:33.857551617 (596h10m33.728634652s) [upload=522603]: Got next segment. Inline: false Dec 30 11:39:33.857584299 (596h10m33.728667334s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:39:33.857589389 (596h10m33.728672424s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:39:34.102965955 (596h10m33.97404899s) [seg_pos={0 0}, upload=522603]: Begin upload segment... Dec 30 11:39:34.103006453 (596h10m33.974089488s) [seg_pos={0 0}, upload=522603]: Done begin upload segment. Dec 30 11:39:34.103029777 (596h10m33.974112812s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103195282 (596h10m33.974278327s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103667051 (596h10m33.974750086s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103912248 (596h10m33.974995283s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104116967 (596h10m33.975200032s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104268255 (596h10m33.975351291s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104427067 (596h10m33.975510112s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104556694 (596h10m33.975639729s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104766183 (596h10m33.975849218s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104863469 (596h10m33.975946504s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105084309 (596h10m33.976167344s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105175323 (596h10m33.976258368s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105500431 (596h10m33.976583466s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105759916 (596h10m33.976842951s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106158264 (596h10m33.977241299s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106444299 (596h10m33.977527334s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:35.810341633 (596h10m35.681424668s) [upload=522603]: Next returned nil segment Dec 30 11:39:35.810481680 (596h10m35.681564726s) [upload=522603]: Waiting for error group managing segments... Dec 30 11:39:36.089501399 (596h10m35.960584434s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.092835126 (596h10m35.963918161s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.097761046 (596h10m35.968844081s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.102058158 (596h10m35.973141193s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.107038902 (596h10m35.978121938s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.114852431 (596h10m35.985935466s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.119241207 (596h10m35.990324242s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121297313 (596h10m35.992380348s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121677617 (596h10m35.992760652s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.178766824 (596h10m36.049849869s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.291087951 (596h10m36.162170997s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.296504245 (596h10m36.167587281s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301077393 (596h10m36.172160428s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301098233 (596h10m36.172181268s) [seg_pos={0 0}, upload=522603]: Segment reached optimal threshold of 13 pieces. Dec 30 11:39:36.301289016 (596h10m36.172372061s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301301980 (596h10m36.172385045s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301453720 (596h10m36.172536765s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301490470 (596h10m36.172573505s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301648460 (596h10m36.172731495s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301656837 (596h10m36.172739872s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301671514 (596h10m36.172754559s) [seg_pos={0 0}, upload=522603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:39:36.301740436 (596h10m36.172823471s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:39:36.301745044 (596h10m36.172828079s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:39:36.591390145 (596h10m36.46247319s) [upload=522603]: Done uploading segments... Dec 30 11:39:36.591416415 (596h10m36.46249946s) [upload=522603]: Upload finished. err: <nil> Duration: 5.786169512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X..........X...X...........X....X.X........X.....] duration: 40.837441326s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 40.837461434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X.] duration: 3.692103286s trace: https://jaeger.dev.storj.io/trace/3aa40edc5615884b FAIL (2025-12-30 11:40:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.692128284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..X] duration: 5.001151795s trace: https://jaeger.dev.storj.io/trace/3aa40edc5615884b FAIL (2025-12-30 11:40:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001175862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X..........X....X.......X.....X..XXX....XX.X.X.X........X] duration: 30.007514126s trace: https://jaeger.dev.storj.io/trace/5675c8f3d22b61db FAIL (2025-12-30 11:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ) context deadline exceeded ======LOGS====== Dec 30 11:39:33.857167265 (596h10m33.7282503s) [upload=522603]: Starting upload Dec 30 11:39:33.857244021 (596h10m33.728327086s) [upload=522603]: Done starting upload Dec 30 11:39:33.857390640 (596h10m33.728473675s) [upload=522603]: Uploading segments... Dec 30 11:39:33.857551617 (596h10m33.728634652s) [upload=522603]: Got next segment. Inline: false Dec 30 11:39:33.857584299 (596h10m33.728667334s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:39:33.857589389 (596h10m33.728672424s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:39:34.102965955 (596h10m33.97404899s) [seg_pos={0 0}, upload=522603]: Begin upload segment... Dec 30 11:39:34.103006453 (596h10m33.974089488s) [seg_pos={0 0}, upload=522603]: Done begin upload segment. Dec 30 11:39:34.103029777 (596h10m33.974112812s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103195282 (596h10m33.974278327s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103667051 (596h10m33.974750086s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.103912248 (596h10m33.974995283s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104116967 (596h10m33.975200032s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104268255 (596h10m33.975351291s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104427067 (596h10m33.975510112s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104556694 (596h10m33.975639729s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104766183 (596h10m33.975849218s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.104863469 (596h10m33.975946504s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105084309 (596h10m33.976167344s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105175323 (596h10m33.976258368s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105500431 (596h10m33.976583466s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.105759916 (596h10m33.976842951s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106158264 (596h10m33.977241299s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:34.106444299 (596h10m33.977527334s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Uploading piece... Dec 30 11:39:35.810341633 (596h10m35.681424668s) [upload=522603]: Next returned nil segment Dec 30 11:39:35.810481680 (596h10m35.681564726s) [upload=522603]: Waiting for error group managing segments... Dec 30 11:39:36.089501399 (596h10m35.960584434s) [noise=true , address=145.79.227.4:4185 , piece_id=TGEKKESVVYY47GQTEOHW3GMM75YH5VOVXJBGESTZDNLKLCJFNOWA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.092835126 (596h10m35.963918161s) [noise=true , address=145.79.227.6:4258 , piece_id=3COOPSQPJEQZB6NAD3VURQO5SHVAUKXBLQ3BH7L6ECXSCSBPEH5Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.097761046 (596h10m35.968844081s) [noise=true , address=145.79.227.14:4839 , piece_id=CIYONYEYR7P5WAPVRILVR465GXJSHRQRMKJOENSRJWMXIN2BBD5A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.102058158 (596h10m35.973141193s) [noise=true , address=145.79.227.19:4078 , piece_id=VQ7DCILESYR3RE4GRPRTBCYQBUJXJWYN2JK4F3GMWP5E2EVBML2A, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.107038902 (596h10m35.978121938s) [noise=true , address=145.79.227.45:4230 , piece_id=R4HDTSS6C2SBXSQTQSQ4SM6WPER5VJO3YDDMVCRGIYOK52OMX6KA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.114852431 (596h10m35.985935466s) [noise=true , address=145.79.227.42:4122 , piece_id=EVC3E5ZAQUEMXXLP2ZFRPOM2QPEQGMRAXPGPHETDA76ZEVCVIFUA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.119241207 (596h10m35.990324242s) [noise=true , address=145.79.227.26:4653 , piece_id=OELFEPPUAL3GD4JBJPTHXBXRPTVMF7ZBW6EEKHBB4NEI7ZIVPWFQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121297313 (596h10m35.992380348s) [noise=true , address=145.79.227.36:4517 , piece_id=U7UBINOB3D5PRQAXWXMTI235V5YNNVHCJWMHFEYM36KFEBDOWOSA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.121677617 (596h10m35.992760652s) [noise=true , address=145.79.227.37:4567 , piece_id=GE6F6DCLNS7ESA3T73FN4VSM4IULFA2WNDYGD2NITJFOTS2KT32Q, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.178766824 (596h10m36.049849869s) [noise=true , address=145.79.227.32:4834 , piece_id=PWLO7KQGKECK7NUPBIDJOTQ4LYAGKKHSKTGHBT5EKCOXW5UHSQJQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.291087951 (596h10m36.162170997s) [noise=true , address=77.90.31.17:4770 , piece_id=TVDWFR7TOSR7SCOEDCMWZTDUJ5SMCBACTPRRR5ITIMB5X5XJSCOQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.296504245 (596h10m36.167587281s) [noise=true , address=77.90.31.18:4808 , piece_id=MDABPB54OTXJ6CUXUIKI5NWPGFDXJHGYZHNLKJ25VEK2E5FF6GEQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301077393 (596h10m36.172160428s) [noise=true , address=77.90.31.19:4841 , piece_id=XJIMJ36RN6K324FFUWQGJI2H3BA2YMRNKXMXGE5WCGAZAEO7JVDA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: <nil> Dec 30 11:39:36.301098233 (596h10m36.172181268s) [seg_pos={0 0}, upload=522603]: Segment reached optimal threshold of 13 pieces. Dec 30 11:39:36.301289016 (596h10m36.172372061s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301301980 (596h10m36.172385045s) [noise=true , address=45.91.116.13:4156 , piece_id=QHFPPOWUWMQXSGR7232KZ76XE42L5SY72WJPBTD5FKIJA4VZJXGQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301453720 (596h10m36.172536765s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301490470 (596h10m36.172573505s) [noise=true , address=77.90.31.16:4735 , piece_id=TTA4SCQSXLBKIPSZZI6YTM7S6HFAIQRFXIQFIPLBQFRI6PIEUUBA, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301648460 (596h10m36.172731495s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 30 11:39:36.301656837 (596h10m36.172739872s) [noise=true , address=145.79.227.30:4761 , piece_id=BISOHTA5VT7VUZCHI6FBFBOG4TLU3LYYNKP22IMSGM2LPNKHVITQ, seg_pos={0 0}, upload=522603]: Long tail cancellation due to optimal threshold Dec 30 11:39:36.301671514 (596h10m36.172754559s) [seg_pos={0 0}, upload=522603]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:39:36.301740436 (596h10m36.172823471s) [upload=522603]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:39:36.301745044 (596h10m36.172828079s) [upload=522603]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:39:36.591390145 (596h10m36.46247319s) [upload=522603]: Done uploading segments... Dec 30 11:39:36.591416415 (596h10m36.46249946s) [upload=522603]: Upload finished. err: <nil> Duration: 30.007541579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X..........X...X...........X....X.X........X....] duration: 35.707028967s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 35.707055507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [......X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X..] duration: 3.638424183s trace: https://jaeger.dev.storj.io/trace/1b35662c1c4246d FAIL (2025-12-30 11:39:31 UTC) orbiter warning: context deadline exceeded Duration: 3.638454211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X..........X...X...........X....X.X........X...] duration: 53.657280109s trace: https://jaeger.dev.storj.io/trace/40a87f21171f227 FAIL (2025-12-30 11:39:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=522543]: Uploading piece... Dec 30 11:35:24.110455266 (596h6m23.981538311s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.110524468 (596h6m23.981607513s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111022496 (596h6m23.982105541s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111321446 (596h6m23.982404491s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111442195 (596h6m23.98252524s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111791761 (596h6m23.982874796s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.111889337 (596h6m23.982972372s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112252048 (596h6m23.983335083s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112350895 (596h6m23.98343394s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.112679702 (596h6m23.983762737s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.113173161 (596h6m23.984256196s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.121781242 (596h6m23.992864287s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.123474747 (596h6m23.994557793s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.125792081 (596h6m23.996875126s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.127054634 (596h6m23.998137679s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.128850594 (596h6m23.999933629s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:24.134832835 (596h6m24.00591588s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Uploading piece... Dec 30 11:35:25.570961836 (596h6m25.442044871s) [upload=522543]: Next returned nil segment Dec 30 11:35:25.570979930 (596h6m25.442062975s) [upload=522543]: Waiting for error group managing segments... Dec 30 11:35:25.969548128 (596h6m25.840631163s) [noise=true , address=145.79.227.6:4283 , piece_id=XVKXVYQWJGFPLOMEHIXSU7IHJUC4SKS6A6LGPNF7KINZVNRR27XQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.977055464 (596h6m25.848138509s) [noise=true , address=145.79.227.21:4495 , piece_id=IETMPN4PDFAK4O3YME3CVGA4573DG3Q73OLWAMTUHFJZTCGWURGQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:25.994339906 (596h6m25.865422941s) [noise=true , address=145.79.227.43:4169 , piece_id=PDEVBAK35CB3Z2KTE6K7AU6B5PNZDOJ4KJBOE7DST3U3C5IRP2CQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.150853292 (596h6m26.021936327s) [noise=true , address=77.90.31.11:4583 , piece_id=P645DCP5KFG2KY6BRLQNWIMB6Y66CHVBJVTJIDKQFDE4EJZHDB5A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.170999484 (596h6m26.042082519s) [noise=true , address=77.90.31.19:4868 , piece_id=AYEKBEOZ2LLQBYDFEZRIVW7REERSQVTG44WUHWB3D7OP3BTXQY4Q, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.340603598 (596h6m26.211686633s) [noise=true , address=45.91.116.5:4319 , piece_id=I4QUNFOYQ7HILLEQBXIKDLXFAPJ2SJJO3AXEP5G6AB3KAJ2GVRLQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.571674474 (596h6m26.442757509s) [noise=true , address=77.90.31.10:4528 , piece_id=A6YZKZRZPLMIJLDJFWW6ST6SXYJISJQD26EMCVU35646GLEFJVEA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.581533887 (596h6m26.452616932s) [noise=true , address=77.90.31.17:4800 , piece_id=EGJZA6BXTTKONRWEMMOPTMOZIM6ZS26T3576NNEG2T5P52KYBNQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.872378452 (596h6m26.743461497s) [noise=true , address=145.79.227.30:4775 , piece_id=NWDANRHE6IVLWFV56OGPHB2RQLC5GLSLHGWX2R3QU2KRCYBP2MKQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.927432423 (596h6m26.798515468s) [noise=true , address=64.49.13.27:4963 , piece_id=QOD77Q5HCXYJV6P5PWZP2CYHEBPPX4J6FERRMQNZ7SG2GJ3TGG4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: ecclient: upload failed (node:1n6MUXcUQ7sT61AwvkhYRq4wniRMQDWMfFk1tENVSnUiDx38rq, address:64.49.13.27:4963): protocol: storage node overloaded, request limit: 2000 Dec 30 11:35:26.928640775 (596h6m26.79972382s) [noise=true , address=145.79.227.40:4067 , piece_id=NGFT4HSQPA3CZSCDZYNRHIAX336EPHSWJN6BCH3SSELXSJLKKI4A, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.942047663 (596h6m26.813130699s) [noise=true , address=145.79.227.33:4898 , piece_id=RBM7Z7O3OLBQBKRC3FBJ6KWBLNC5RTCBMRZXWCOLYUY4HQHIMKWQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:26.965070583 (596h6m26.836153618s) [noise=true , address=145.79.227.44:4225 , piece_id=TKLQBIPYFZUXYAPXCPOWGJFOXNDF5AHYC5TLFTPZI3X3HDZHAYQQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.003657148 (596h6m26.874740183s) [noise=true , address=145.79.227.34:4931 , piece_id=5NEZCHIQTEIZ6TOEXOCPUIRBYQBILJ26EY5GEDX4ZJE3MCQMQ2KQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012392531 (596h6m26.883475567s) [noise=true , address=145.79.227.49:4350 , piece_id=BYWXYLZD5375A2TEWCPQXPQABMA6CHB57EKENVD5ZI3JSCHFQZUQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: <nil> Dec 30 11:35:27.012414193 (596h6m26.883497228s) [seg_pos={0 0}, upload=522543]: Segment reached optimal threshold of 14 pieces. Dec 30 11:35:27.012641886 (596h6m26.883724931s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012652347 (596h6m26.883735382s) [noise=true , address=145.79.227.35:4513 , piece_id=Y44NUJS5RGNO2VBLFLZEX6X7TXXYF544JVIIVO6EZYZZBMPYDGUA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012808995 (596h6m26.88389203s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60670->64.49.13.169:4715: use of closed network connection Dec 30 11:35:27.012816418 (596h6m26.883899454s) [noise=true , address=64.49.13.169:4715 , piece_id=WNPGNR2ICQLGZUUO73RS62LBREHFBWIA6SLF57ICNBRXU36JOOFA, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012936578 (596h6m26.884019623s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy): protocol: expected piece hash; context canceled; EOF Dec 30 11:35:27.012942038 (596h6m26.884025073s) [noise=true , address=193.149.164.143:4696 , piece_id=NJ6OWCMNVKSBTFWELLBYBYPCWI34UGZ5LIDC6XSCE65YOZXXIIMQ, seg_pos={0 0}, upload=522543]: Long tail cancellation due to optimal threshold Dec 30 11:35:27.012959701 (596h6m26.884042736s) [seg_pos={0 0}, upload=522543]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:35:27.013022441 (596h6m26.884105486s) [upload=522543]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:35:27.013026518 (596h6m26.884109554s) [upload=522543]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:35:27.300012966 (596h6m27.171096011s) [upload=522543]: Done uploading segments... Dec 30 11:35:27.300033526 (596h6m27.171116561s) [upload=522543]: Upload finished. err: <nil> Duration: 53.657297672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.......X.X.XX.X.X.X.X...X.XX..XXX.X..XX.......X..XX.X...X.X.] duration: 5.00124314s trace: https://jaeger.dev.storj.io/trace/1b35662c1c4246d FAIL (2025-12-30 11:38:48 UTC) orbiter warning: context deadline exceeded Duration: 5.001275713s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)