Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 2.13157155s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:03:47 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.131594143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 17.095164647s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:03:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 17.095192331s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 3.630025332s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:03:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 3.630053827s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X..........X......X....X...........X...X.X..XX............] duration: 3.478502148s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:03:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478526885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...............X...........................................] duration: 5.999330708s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:03:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.999356648s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.340620222s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:03:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.340647895s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 735.169236ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:03:18 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 735.20279ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.942729049s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:03:15 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.942757244s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X...................X.................X...........] duration: 5.836219006s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.836245605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................X.................X..........] duration: 5.93160658s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:02:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.931770763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X......X..............................................] duration: 2.113822243s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:02:42 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.113855725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.547451118s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:02:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.547522734s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 913.174799ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:02:41 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 913.205749ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...............X..........................................] duration: 6.920945628s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:02:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.920975505s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X..........X......X....X...........X...X.X..XX...........] duration: 3.516678276s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:02:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516705788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.263150158s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:02:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.263182409s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.003450862s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:02:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.003483685s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X..........X......X....X...........X...X.X..XX..........] duration: 3.528062885s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:02:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528092632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.290178257s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:02:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.290208404s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration: 2.077954159s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:01:56 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.077995596s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.106538631s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:01:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.106566144s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X...................X.................X.........] duration: 5.952698385s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:01:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.952731058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X......X............................................] duration: 2.888575345s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:01:40 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.888607047s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.127803154s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:01:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.127893396s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...............X.........................................] duration: 7.157403893s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:01:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.157431786s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.751004552s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:01:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.751044027s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.849660673s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:01:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.849689296s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X..........X......X....X...........X...X.X..XX.........] duration: 3.357128087s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:01:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.357156411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 2.243593452s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:01:01 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.243623951s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.858243627s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:01:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.858277491s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.796547519s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:00:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.796595371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X..........X......X....X...........X...X.X..XX........] duration: 3.4305122s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:00:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430528862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.018634027s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:00:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.018651589s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...............X........................................] duration: 6.798273496s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:00:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.798300898s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 694.475472ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:00:43 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.503896ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X...................X.................X........] duration: 5.745018583s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:00:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.745028693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.675067226s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:00:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.675100709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X......X..........................................] duration: 2.310900746s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:00:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.310932898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...............X.......................................] duration: 5.753756123s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:00:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.753788244s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 705.608954ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:00:15 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 705.641727ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.063934452s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:00:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.063971953s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X..........X......X....X...........X...X.X..XX.......] duration: 3.422109109s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422142241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.794824419s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:59:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.794853964s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.036462776s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 03:59:51 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.036542869s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...............X......................................] duration: 6.645422323s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 03:59:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.645530369s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.737494304s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 03:59:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.737524451s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................X.................X.......] duration: 5.679847779s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 03:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.679872665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.760043956s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 03:59:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.760069215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X......X.........................................] duration: 2.051236898s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 03:59:23 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.051264741s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.848631522s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 03:59:22 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.848656972s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)