Orbiter History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 802.181379ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:09:07 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 802.23444ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X...................X.................X...............X...] duration: 6.115788685s trace: https://jaeger.dev.storj.io/trace/194dcfc888c86a7 FAIL (2025-12-08 04:09:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.115821528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 7.06823142s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:09:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.068262709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.331176785s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:08:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.331205809s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.205173692s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:08:58 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.205196165s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X......X....X...........X...X.X..XX.............X....X] duration: 5.000402647s trace: https://jaeger.dev.storj.io/trace/397ed9910501d7e9 FAIL (2025-12-08 04:08:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000429067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X...................X.................X...............X..] duration: 5.843828223s trace: https://jaeger.dev.storj.io/trace/194dcfc888c86a7 FAIL (2025-12-08 04:08:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.843861656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.130360118s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:08:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 3.130387881s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.162472357s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:08:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.162502615s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X......X....X...........X...X.X..XX.............X....] duration: 3.445440788s trace: https://jaeger.dev.storj.io/trace/6d6f90ce34b5cd31 FAIL (2025-12-08 04:08:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445460244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.05652108s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:08:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.056548963s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X...................X.................X...............X.] duration: 5.781732351s trace: https://jaeger.dev.storj.io/trace/194dcfc888c86a7 FAIL (2025-12-08 04:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.781752399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.073675371s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:08:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.073719134s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X...................X.................X...............X] duration: 1m0.00138114s trace: https://jaeger.dev.storj.io/trace/194dcfc888c86a7 FAIL (2025-12-08 04:08:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001436976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X......X....................................................] duration: 2.082505169s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:07:59 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.082538683s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 701.08876ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:07:58 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 701.1147ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.075610079s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:07:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.075630217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X......X....X...........X...X.X..XX.............X...] duration: 3.545584248s trace: https://jaeger.dev.storj.io/trace/6d6f90ce34b5cd31 FAIL (2025-12-08 04:07:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545610248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.164316909s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:07:31 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.164350984s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 694.40326ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:07:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.493643ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.201978817s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:07:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.201997073s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........X......X....X...........X...X.X..XX.............X..] duration: 3.478356193s trace: https://jaeger.dev.storj.io/trace/6d6f90ce34b5cd31 FAIL (2025-12-08 04:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478459811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.429208165s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:06:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.42925269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.979949791s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:06:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.979976604s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X......X...................................................] duration: 2.411366601s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:06:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.411395346s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.840831142s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:06:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 3.840864897s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X...................X.................X...............] duration: 5.739871523s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.73997476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.963918068s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:06:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.963944638s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................X.................X..............] duration: 5.864735779s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:05:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86476173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..........X......X....X...........X...X.X..XX.............X.] duration: 3.510079561s trace: https://jaeger.dev.storj.io/trace/6d6f90ce34b5cd31 FAIL (2025-12-08 04:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510108857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.94382504s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:05:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.943855117s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 988.928027ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:05:45 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 988.95622ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 7.207326886s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:05:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.207355009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.860628319s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.860772824s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.014118321s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:05:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.014153638s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X......X..................................................] duration: 2.045138631s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:05:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.04515916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.08512262s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:05:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.085149252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.95844002s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:05:02 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.958479445s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.090379149s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:05:01 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.090407643s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X...................X.................X.............] duration: 5.753668454s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:04:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.753698391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X..........X......X....X...........X...X.X..XX.............X] duration: 5.001127548s trace: https://jaeger.dev.storj.io/trace/6d6f90ce34b5cd31 FAIL (2025-12-08 04:04:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001148468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X......X.................................................] duration: 2.709227395s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:04:44 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.709252182s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.727858856s trace: https://jaeger.dev.storj.io/trace/5ce16d59ac7ca3 FAIL (2025-12-08 04:04:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== V3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478633206 (59h44m15.349716251s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.478935492 (59h44m15.350018537s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479212750 (59h44m15.350295785s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.479658087 (59h44m15.350741132s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.480729167 (59h44m15.351812212s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481358675 (59h44m15.35244171s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.481870881 (59h44m15.352953916s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482011018 (59h44m15.353094063s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.482599078 (59h44m15.353682113s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:15.483043735 (59h44m15.35412679s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Uploading piece... Dec 8 03:13:16.296121260 (59h44m16.167204295s) [upload=52461]: Next returned nil segment Dec 8 03:13:16.296136789 (59h44m16.167219824s) [upload=52461]: Waiting for error group managing segments... Dec 8 03:13:16.688782787 (59h44m16.559865822s) [noise=true , address=185.167.204.206:4059 , piece_id=G4AB6LYXSDKJOYKXNAILU34CEGNPXWYJQICBGHLDTWNH3TMHA7JA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.689199911 (59h44m16.560282946s) [noise=true , address=185.167.204.206:4043 , piece_id=PEASXJDAI2ORI5ZOH5GMCNMFNXKL2SKKZP3WBQ72VOGB7BJZKWHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690209955 (59h44m16.561293s) [noise=true , address=185.167.204.206:4055 , piece_id=RN3ETZZORZJ5FLRZWXGDTBOJP2ACTVIXO7QSVRH4EY7H4HQNZIKA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.690460382 (59h44m16.561543427s) [noise=true , address=185.167.204.205:4010 , piece_id=5TOGC4H52AGJS6ZZJJM56DZXZQJ2FBK2YZO6634ACFVLFGOOHC5A, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.696363612 (59h44m16.567446657s) [noise=true , address=185.167.204.206:4051 , piece_id=SPX6V5WZAWUFRZ23KZKQPPWS3VX37O6EOOQTCDRXXNMIGQGSERMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697250752 (59h44m16.568333787s) [noise=true , address=185.167.204.205:4011 , piece_id=ZM5KFXNZTZTU6WS33BQ4HHO2G4EHQGNCZKVNSKXPXEXAM4RW6BQA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.697962948 (59h44m16.569045983s) [noise=true , address=185.167.204.206:4072 , piece_id=F5EN43FGL2LXIZU4KHKDGCXM3O7YJM37AT2XBHRLSEOWDJMP5GCA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.700404106 (59h44m16.571487151s) [noise=true , address=185.167.204.206:4050 , piece_id=ZA5QCT6A7AZJL25YXEPHFFBMLXGMIV37WVYZKN6IBGIUYO4UKCDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701120280 (59h44m16.572203315s) [noise=true , address=185.167.204.206:4058 , piece_id=ODSLWPR4JMPSNQSUL3OHUAUKBDCUTUX2UGH2WZDPE7PBOW3NBA3Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.701632446 (59h44m16.572715481s) [noise=true , address=185.167.204.206:4040 , piece_id=ORW7CUV65QLBVVH2SNJRBLINACXMI4C6HXYSMWFUZ7OUAZX5YDPA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.702967407 (59h44m16.574050453s) [noise=true , address=185.167.204.205:4019 , piece_id=URV3YKXL43AUX3HQ4QJ62BOB7R2A55YYYREHC645WOGLRCHEROOA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.706243235 (59h44m16.57732628s) [noise=true , address=185.167.204.205:4035 , piece_id=6NZETOEX253U2AGIILFBIK3JGBL2S7PJV6WE77ITLVEROV6SZVWA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.710948004 (59h44m16.582031049s) [noise=true , address=185.167.204.206:4061 , piece_id=MKP6KY35F7BXGDRUFVWEI6Y7OJXYSIX45452625SPQVGRT4X5GHQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:16.712808888 (59h44m16.583891933s) [noise=true , address=185.167.204.206:4065 , piece_id=EPW6FRXQZKBOVMTROQYK5O33BBZHIW7BU5HMHKPMOWZNICQJWWMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.193700893 (59h44m17.064783928s) [noise=true , address=81.171.14.70:4031 , piece_id=KPTJZ4FYTXEYVVSLRQD42OL7NGOTXQF5RVPA3PTC4L7W5AZWQNMQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.205494490 (59h44m17.076577535s) [noise=true , address=81.171.14.70:4026 , piece_id=UQBSQNBMHZY7733ZI6VNDEU5WXZ2GUBRWPUL3W5LRLWEVLXFN7GQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.213382981 (59h44m17.084466026s) [noise=true , address=81.171.14.70:4032 , piece_id=KJAHFSUBCJ5RSF3R5VSI5M2U7C7AGSE6P76P253SDQT7SBRZ2ZSQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.223902893 (59h44m17.094985938s) [noise=true , address=81.171.14.70:4042 , piece_id=LNABALIBR2GPDGMNG6HKTKLBPSQH3HB5HX67MCMY6LEUV22HX2IA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.272328052 (59h44m17.143411088s) [noise=true , address=81.171.14.70:4025 , piece_id=XGAQ5L2ATTPBNPOA45SCASWNN6JT7L5K5TB5IQSAYLM4QUZQX2MQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.314279466 (59h44m17.185362501s) [noise=true , address=81.171.14.70:4038 , piece_id=QWZFMCB3BKGSROA6WTKBMTIKIKZ744UZ7222NGF2ZRMXMJH55K2Q, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.339184389 (59h44m17.210267434s) [noise=true , address=81.171.14.70:4029 , piece_id=N2MJLXIIMGZZG3TCJ5LJUGH4J5H5OXA6GOGLSBTN6BEMDXOWGTDA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.352328829 (59h44m17.223411864s) [noise=true , address=81.171.14.70:4028 , piece_id=K4GHMGJRVY5233RXSDQSHW4OQSXK6A534J36KNIJZNRBLE2YMNOQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.363261126 (59h44m17.234344171s) [noise=true , address=81.171.14.70:4037 , piece_id=5Q5SC2XQKNY5V26DWVUQXYJJZQE2JXH5RWJWBC4MCGDQYO5ZBIMA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.770690916 (59h44m17.641773951s) [noise=true , address=81.171.14.70:4027 , piece_id=SRP3XCTIO67QDY2APAFNUGDVJYP4NTNH2ALVZO3CKQWJRGONSDHA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:17.980125476 (59h44m17.851208521s) [noise=true , address=89.149.201.147:4005 , piece_id=DLFODTUO7RTGVF5CKBB5NFBCRLEVBR57I2W746O6JQX4ZJ6Q4RUA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040367989 (59h44m17.911451024s) [noise=true , address=89.149.201.147:4007 , piece_id=33HGXVE4ZG2EH6ROJSXS2WRSZSM3XFH7JXLTHT6QOIADMWWMB4SA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: <nil> Dec 8 03:13:18.040388418 (59h44m17.911471453s) [seg_pos={0 0}, upload=52461]: Segment reached optimal threshold of 26 pieces. Dec 8 03:13:18.040618335 (59h44m17.91170137s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RDCCUcFecPi2tRuk69SueE84EdSssocDPB4WEmUQaXGacDAhc): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040631010 (59h44m17.911714055s) [noise=true , address=89.149.201.147:4002 , piece_id=EXQCZSW5RDE3KJTHWTFJ5GS24FKBKTELCIHD7NPUAPAAF2EWZ6CA, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040763592 (59h44m17.911846647s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 8 03:13:18.040782438 (59h44m17.911865483s) [noise=true , address=89.149.201.147:4021 , piece_id=IGHIV22JPHXISR23YX4PVYDMMFKQM266YOHZ3F37A3PUCU4LRCEQ, seg_pos={0 0}, upload=52461]: Long tail cancellation due to optimal threshold Dec 8 03:13:18.040832964 (59h44m17.911916009s) [seg_pos={0 0}, upload=52461]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 03:13:18.040964104 (59h44m17.912047149s) [upload=52461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:13:18.040993821 (59h44m17.912076866s) [upload=52461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:25:31.019642569 (59h56m30.890725614s) [upload=52461]: Done uploading segments... Dec 8 03:25:31.019743331 (59h56m30.890826366s) [upload=52461]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.727891669s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.49984887s trace: https://jaeger.dev.storj.io/trace/15c53885b53c2c0b FAIL (2025-12-08 04:04:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.499878366s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 2.118166073s trace: https://jaeger.dev.storj.io/trace/67981d3422e72759 FAIL (2025-12-08 04:04:26 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.118193616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X..........X......X....X...........X...X.X..XX.............] duration: 3.402683757s trace: https://jaeger.dev.storj.io/trace/556a23bce4f7722c FAIL (2025-12-08 04:04:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402709979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.417334668s trace: https://jaeger.dev.storj.io/trace/181aeb075a359abb FAIL (2025-12-08 04:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454589606 (59h45m50.325672641s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.454901751 (59h45m50.325984786s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455220038 (59h45m50.326303083s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455621522 (59h45m50.326704557s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.455956660 (59h45m50.327039695s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456289073 (59h45m50.327372118s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456612098 (59h45m50.327695144s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.456946765 (59h45m50.32802981s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.458680467 (59h45m50.329763512s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.460060705 (59h45m50.33114374s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.461350561 (59h45m50.332433606s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.462929117 (59h45m50.334012162s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.464669651 (59h45m50.335752696s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466275188 (59h45m50.337358223s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.466899267 (59h45m50.337982312s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:50.468267071 (59h45m50.339350106s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Uploading piece... Dec 8 03:14:51.824746486 (59h45m51.695829522s) [upload=52483]: Next returned nil segment Dec 8 03:14:51.824761806 (59h45m51.695844881s) [upload=52483]: Waiting for error group managing segments... Dec 8 03:14:52.168683776 (59h45m52.039766811s) [noise=true , address=64.49.13.139:4479 , piece_id=NDIMXSB7HI2275EIRSZXF7RMLAK7Z4CHIY3KV4FQOW3W3ODNFHCQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.219276294 (59h45m52.090359339s) [noise=true , address=64.49.13.137:4503 , piece_id=XRE3OC6EOYKBSVH5QSJEP3OEMABQICG757PBJF4BO6Y2NZXKLUCA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.232504452 (59h45m52.103587498s) [noise=true , address=64.49.13.138:4484 , piece_id=B662IVDCZXDQD4CRWJWIO2RKJ4FSWOJPXPSJMJP57S7HJZPMWNFQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.359951265 (59h45m52.23103431s) [noise=true , address=77.90.31.19:4845 , piece_id=Y6O57T42S6MJ6OIVVC6RRM7HHUBDAL2IZIIVXSX2ATWEJOLCDJ6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.430948747 (59h45m52.302031782s) [noise=true , address=77.90.31.13:4652 , piece_id=NJFD2U6JYML2P6FVDQHWGCKETKK2DMA7H67RKVMHOTP7FAWJOG6Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442354135 (59h45m52.31343718s) [noise=true , address=77.90.31.12:4609 , piece_id=5JSQXATSAR7IUISI5H2BVLIAWKA7J3NQDYGD4WU5OQAJ3DYTQKZA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.442770528 (59h45m52.313853573s) [noise=true , address=45.91.116.11:4147 , piece_id=JSBISTRA2PFSSBP77R4MYQLMGLSXK37AK5SZWMJAAJXPEPFSZJXA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444206031 (59h45m52.315289076s) [noise=true , address=145.79.227.48:4325 , piece_id=L73GCV6XNO7U4RYUJQ7LDO77ZXRII53HEDSJ3Q5RTDC25NZEJJMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.444711955 (59h45m52.31579499s) [noise=true , address=145.79.227.45:4237 , piece_id=UGE4PKKPY3ZPQNEVY737USGNXPFU2D6UCGR4E75UMMG5H272HJPQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.454853546 (59h45m52.325936581s) [noise=true , address=145.79.227.40:4068 , piece_id=NPBXOCU6KDOOBAA3TFZOUQMX6YX4C2EEMEJJF6U4FSYY5ER7XNJQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.459856934 (59h45m52.330939979s) [noise=true , address=45.91.116.5:4331 , piece_id=TRK7QDEYWWDD4JYB3KLIM63YGXPJCXDCMZBN5XGQIUQSJYTRMO6A, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.469367493 (59h45m52.340450528s) [noise=true , address=45.91.116.4:4969 , piece_id=ERZVYCMZ354H764OBTGAJJLFZIXMT6VPVICU3SRJXFQXHJSTCOPA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.479951517 (59h45m52.351034562s) [noise=true , address=45.91.116.13:4169 , piece_id=LKY7DXT35POC65VJ266C7BHUH7FIDS4Y3EFSQDLP5X6THU4HHHJA, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.515994289 (59h45m52.387077324s) [noise=true , address=45.91.116.6:4998 , piece_id=3PM72VJIDSFQGE5AK3ISSI2ZURYTMBYDHH6DJN37WZZCG3RCWC5Q, seg_pos={0 0}, upload=52483]: Done uploading piece. err: <nil> Dec 8 03:14:52.516015168 (59h45m52.387098203s) [seg_pos={0 0}, upload=52483]: Segment reached optimal threshold of 14 pieces. Dec 8 03:14:52.516331542 (59h45m52.387414577s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaVAZApAbM17VQYWfZFhWEvsKSrKWHwWiZmG1eaZgZd5iYEKR): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516340358 (59h45m52.387423393s) [noise=true , address=145.79.227.32:4844 , piece_id=ZRXV7BROKDF33HMEI2TPMIY4AVE4IDZH2BKE6MVYDMEF4BM4ZXQQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516550478 (59h45m52.387633513s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516557191 (59h45m52.387640226s) [noise=true , address=77.90.31.16:4759 , piece_id=QXTY3KYIA2AW2NFXAAC4SFODVGDFVW2MKPQJXKGCM5FOF3WX77MQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516655238 (59h45m52.387738273s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 8 03:14:52.516661650 (59h45m52.387744685s) [noise=true , address=145.79.227.17:4990 , piece_id=D2X4UEGUYN756GRJRTYRPCAM6PQUJT3K6KFYJO4WVD2SWQDGVEUQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516879334 (59h45m52.387962379s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ): protocol: read tcp 188.166.225.170:54622->145.79.227.12:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 03:14:52.516887290 (59h45m52.387970335s) [noise=true , address=145.79.227.12:4782 , piece_id=YEG3OATW7OFRQKG3PVIG43BYFTLJECFUDN4VIVZMOF5UJD6MLYMQ, seg_pos={0 0}, upload=52483]: Long tail cancellation due to optimal threshold Dec 8 03:14:52.516909502 (59h45m52.387992537s) [seg_pos={0 0}, upload=52483]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 03:14:52.517012949 (59h45m52.388095984s) [upload=52483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 03:14:52.517018219 (59h45m52.388101264s) [upload=52483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 03:26:03.308190575 (59h57m3.17927362s) [upload=52483]: Done uploading segments... Dec 8 03:26:03.308267711 (59h57m3.179350766s) [upload=52483]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.417394682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................X.................X............] duration: 5.919046381s trace: https://jaeger.dev.storj.io/trace/74a516d113126437 FAIL (2025-12-08 04:04:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.919075489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.908519146s trace: https://jaeger.dev.storj.io/trace/456cfc131726c52f FAIL (2025-12-08 04:04:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.9085478s core uplink eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.068255328s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6efc88c1a2a7b7f2 FAIL (2025-12-08 04:03:54 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.068284814s core satellite us1

Orbiter Check History

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)