Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.067476748s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:35:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.067508619s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.................X.....X...X.X...........X..........X.....] duration: 3.491155531s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491190599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.178400276s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:35:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.178434582s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.................X.....X...X.X...........X..........X....] duration: 3.497838179s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:34:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497903244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X......X.............] duration: 2.023813101s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:34:54 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.023841605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.865295868s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:34:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.865337827s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................X.........X...............X...........] duration: 5.590390829s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:34:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.590506139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X......X............] duration: 2.58973058s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:34:16 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.589765336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 18.646515471s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:34:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.646618417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.................X.....X...X.X...........X..........X...] duration: 3.479882216s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:34:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479899979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.711782333s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:34:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.711808352s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................X...............X.....] duration: 9.521019179s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:34:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.5210461s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.075692855s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:34:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.075725487s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X......X...........] duration: 3.000367845s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:33:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 3.000452195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.........X...............X..........] duration: 5.917032083s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:33:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.917058604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.84972408s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:33:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.849744949s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.67648123s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:33:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.676511587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.087914197s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:33:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.088021581s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................X...............X....] duration: 5.823776209s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:33:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.823803901s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.833305013s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:33:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.833335252s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.................X.....X...X.X...........X..........X..] duration: 3.512486611s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:33:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512515816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.222190189s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.222221358s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................X...............X...] duration: 7.482144747s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:32:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.482174284s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X.........X...............X.........] duration: 5.631443262s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:32:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.631561867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 2.014614425s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:32:46 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.014644982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.94435525s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:32:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.944393342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.089582364s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:32:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.089618673s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X.................X.....X...X.X...........X..........X.] duration: 3.355209066s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:32:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.355240216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.712422583s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:32:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.712460987s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................X...............X..] duration: 6.440026837s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:31:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.440054099s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 2.386988891s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:31:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.38701978s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.........X...............X........] duration: 5.826929814s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:31:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.826958819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.40827135s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:31:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.408298933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.140302742s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:31:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.14033393s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.........X...............X.......] duration: 5.70477028s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:31:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.704794214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.71936663s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:31:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.719398571s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................X...............X.] duration: 5.960100504s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:30:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.960117355s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X.................X.....X...X.X...........X..........X] duration: 5.000816412s trace: https://jaeger.dev.storj.io/trace/2e56da0a6102491c FAIL (2025-12-08 03:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000842872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......................X...............X] duration: 1m36.852130299s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:30:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m36.852150516s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X......X........] duration: 1.992232745s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:30:42 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.99226155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.361579893s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:30:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.361610522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.........X...............X......] duration: 5.716873044s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:30:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.716901419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.177443543s trace: https://jaeger.dev.storj.io/trace/ac7d8769da65842 FAIL (2025-12-08 03:30:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.177475355s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.................X.....X...X.X...........X..........] duration: 3.566601865s trace: https://jaeger.dev.storj.io/trace/796a0d2ea284ddb4 FAIL (2025-12-08 03:30:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566629557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.130445382s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:30:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.130475741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X......X.......] duration: 2.05015935s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:30:08 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.050194886s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.154031844s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:29:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.154177512s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X.........X...............X.....] duration: 5.900034948s trace: https://jaeger.dev.storj.io/trace/178c94d0555094d2 FAIL (2025-12-08 03:29:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.900091616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.664005165s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:29:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.664034841s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 6.314469787s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:29:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 6.314499865s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)