Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 451749 reports (Page 391 of 9035)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................X...X.......................] duration: 5.745224351s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.745249128s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX....] duration: 30.001227036s trace: https://jaeger.dev.storj.io/trace/6a67a37aa13d6714 FAIL (2026-01-11 14:41:29 UTC) orbiter warning: context deadline exceeded Duration: 30.001251672s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.685858166s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:41:09 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2.685882571s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................X...X......................] duration: 5.810556526s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.810583877s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001501743s trace: https://jaeger.dev.storj.io/trace/64386b310477258f FAIL (2026-01-11 14:40:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001524798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX.XX..........X..........XX.XX...XX..X.XX..X.........XX.] duration: 29.620248282s trace: https://jaeger.dev.storj.io/trace/14bf0fa598da3efb FAIL (2026-01-11 14:40:57 UTC) orbiter warning: context deadline exceeded Duration: 29.620262068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.222304353s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:40:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.222331677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X...X.....................] duration: 5.694990604s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:40:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.695018237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX...] duration: 26.26611832s trace: https://jaeger.dev.storj.io/trace/6a67a37aa13d6714 FAIL (2026-01-11 14:40:28 UTC) orbiter warning: context deadline exceeded Duration: 26.266148016s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X.XX.XX..........X..........XX.XX...XX..X.XX..X.........XX] duration: 1m0.001931809s trace: https://jaeger.dev.storj.io/trace/14bf0fa598da3efb FAIL (2026-01-11 14:40:28 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001976265s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.604240955s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:40:27 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 4.604264812s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000832776s trace: https://jaeger.dev.storj.io/trace/4def7e53c23793b2 FAIL (2026-01-11 14:40:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000863355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX..] duration: 26.431799055s trace: https://jaeger.dev.storj.io/trace/6a67a37aa13d6714 FAIL (2026-01-11 14:40:01 UTC) orbiter warning: context deadline exceeded Duration: 26.431843539s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001001936s trace: https://jaeger.dev.storj.io/trace/e78667f7dd1c9cc FAIL (2026-01-11 14:39:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001040119s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 12.007647104s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:39:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.007674176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X...X....................] duration: 5.838751104s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.838779337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...X.XX.XX..........X..........XX.XX...XX..X.XX..X.........X] duration: 1m0.001287735s trace: https://jaeger.dev.storj.io/trace/70e91c98016e608f FAIL (2026-01-11 14:39:27 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001310007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 16.722539292s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:39:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.722562346s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.402563469s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:39:15 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 5.402583357s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX.] duration: 30.000611133s trace: https://jaeger.dev.storj.io/trace/6a67a37aa13d6714 FAIL (2026-01-11 14:39:15 UTC) orbiter warning: context deadline exceeded Duration: 30.00063566s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00064433s trace: https://jaeger.dev.storj.io/trace/f777667bd246bbf FAIL (2026-01-11 14:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000697922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X...X...................] duration: 5.751796041s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:38:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.751820979s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.794185633s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:38:33 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 8.794201935s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000938318s trace: https://jaeger.dev.storj.io/trace/289665fdd8186e6 FAIL (2026-01-11 14:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000994205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 13.436704781s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:38:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.437479947s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.XX] duration: 30.000805848s trace: https://jaeger.dev.storj.io/trace/6a67a37aa13d6714 FAIL (2026-01-11 14:38:08 UTC) orbiter warning: context deadline exceeded Duration: 30.000828972s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [....X.XX.XX..........X..........XX.XX...XX..X.XX..X.........] duration: 35.70845356s trace: https://jaeger.dev.storj.io/trace/2d7b170022a781ae FAIL (2026-01-11 14:38:05 UTC) orbiter warning: context deadline exceeded Duration: 35.708481854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000866107s trace: https://jaeger.dev.storj.io/trace/88e378dbaa0c75e FAIL (2026-01-11 14:37:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001063823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X...X..................] duration: 5.77460317s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:37:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.774626985s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.679761397s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:37:35 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2.679795781s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.X] duration: 30.000850967s trace: https://jaeger.dev.storj.io/trace/484fad22fec4fa05 FAIL (2026-01-11 14:37:25 UTC) orbiter warning: context deadline exceeded Duration: 30.000868642s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.329200671s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:37:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.32921604s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001246989s trace: https://jaeger.dev.storj.io/trace/6b7f02f5e1f5ef6 FAIL (2026-01-11 14:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001261847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.....X.XX.XX..........X..........XX.XX...XX..X.XX..X........] duration: 39.129635303s trace: https://jaeger.dev.storj.io/trace/2d7b170022a781ae FAIL (2026-01-11 14:37:16 UTC) orbiter warning: context deadline exceeded Duration: 39.129662194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001433558s trace: https://jaeger.dev.storj.io/trace/4558e16f59c50457 FAIL (2026-01-11 14:37:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001451732s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.914513548s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:37:04 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1.914541912s edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.531799963s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:36:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 4.531826273s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001071501s trace: https://jaeger.dev.storj.io/trace/436bcd305eca2883 FAIL (2026-01-11 14:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001092602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................X...X.................] duration: 5.936723765s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.936748231s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.254781207s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:36:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.254810983s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000238139s trace: https://jaeger.dev.storj.io/trace/5a467a000e5d84a1 FAIL (2026-01-11 14:36:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000262165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [X..XXXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X.] duration: 23.41915278s trace: https://jaeger.dev.storj.io/trace/1be2b04b0a40aa8d FAIL (2026-01-11 14:36:13 UTC) orbiter warning: context deadline exceeded Duration: 23.419176807s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................X...X................] duration: 5.844028507s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.844056361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.....X.XX.XX..........X..........XX.XX...XX..X.XX..X.......] duration: 28.669657866s trace: https://jaeger.dev.storj.io/trace/2d7b170022a781ae FAIL (2026-01-11 14:36:02 UTC) orbiter warning: context deadline exceeded Duration: 28.669675861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 13.330292471s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:36:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.3303182s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.021343791s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 trace: https://jaeger.dev.storj.io/trace/70855e353f33f12b FAIL (2026-01-11 14:35:34 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 5.021360933s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X.....X.XX.XX..........X..........XX.XX...XX..X.XX..X......] duration: 34.908428311s trace: https://jaeger.dev.storj.io/trace/2d7b170022a781ae FAIL (2026-01-11 14:35:33 UTC) orbiter warning: context deadline exceeded Duration: 34.908463818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................X...X...............] duration: 5.730387876s trace: https://jaeger.dev.storj.io/trace/3408eb41c02279d1 FAIL (2026-01-11 14:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg, Piece ID: 3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ) context deadline exceeded; piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ) context deadline exceeded ======LOGS====== Jan 11 14:25:10.313193802 (886h56m10.184276838s) [upload=776150]: Starting upload Jan 11 14:25:10.313287661 (886h56m10.184370697s) [upload=776150]: Done starting upload Jan 11 14:25:10.313342636 (886h56m10.184425681s) [upload=776150]: Uploading segments... Jan 11 14:25:10.313590928 (886h56m10.184673973s) [upload=776150]: Got next segment. Inline: false Jan 11 14:25:10.313621647 (886h56m10.184704682s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 14:25:10.313626917 (886h56m10.184709952s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 14:25:10.579005031 (886h56m10.450088096s) [seg_pos={0 0}, upload=776150]: Begin upload segment... Jan 11 14:25:10.579061529 (886h56m10.450144564s) [seg_pos={0 0}, upload=776150]: Done begin upload segment. Jan 11 14:25:10.579099030 (886h56m10.450182075s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579164335 (886h56m10.45024738s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.579810344 (886h56m10.450893389s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580371262 (886h56m10.451454307s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.580947500 (886h56m10.452030545s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581198808 (886h56m10.452281863s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581652803 (886h56m10.452735848s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.581700864 (886h56m10.452783929s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582127587 (886h56m10.453210622s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582309283 (886h56m10.453392328s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582456543 (886h56m10.453539588s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582767235 (886h56m10.45385027s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.582941236 (886h56m10.454024271s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583063218 (886h56m10.454146263s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583365634 (886h56m10.454448689s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:10.583601423 (886h56m10.454684458s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Uploading piece... Jan 11 14:25:12.407761809 (886h56m12.278844844s) [upload=776150]: Next returned nil segment Jan 11 14:25:12.407786005 (886h56m12.27886906s) [upload=776150]: Waiting for error group managing segments... Jan 11 14:25:12.698973243 (886h56m12.570056288s) [noise=true , address=77.90.31.14:4663 , piece_id=UTCIRYSPZDDATHUHAK7PQK3GBVFYJMSPJAEXLDPQCXN7CLVU6YCA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.742134127 (886h56m12.613217162s) [noise=true , address=145.79.227.40:4051 , piece_id=SKX5NTIRBEE7JNZEVRFUMMNZRREPTNZCH7NFNVWL4DAERNFTCMGA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.749956642 (886h56m12.621039678s) [noise=true , address=145.79.227.41:4085 , piece_id=XQH557DGIBYLCDI3DN7YI26OKQOOCVQ6B2JLBYTCK7V2YGTIGEBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.800664737 (886h56m12.671747792s) [noise=true , address=77.90.31.21:4877 , piece_id=YVLXYH5Y2BWHMMCNFBGHFE7A5BRK5VIPF4DKS3OL45T7BQ2BIY5A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.805555469 (886h56m12.676638504s) [noise=true , address=145.79.227.17:4971 , piece_id=73SHNGW2BV4HFSE2H7LIAF6I6SMCHZ5WEBCIVXJATPOITFHO56NA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.806742800 (886h56m12.677825835s) [noise=true , address=145.79.227.9:4367 , piece_id=L7DN3ASQTM6ZGO72ZKTRDBKEA34FAVVGRLK75V33ZFAYOY26YKZA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.825678456 (886h56m12.696761491s) [noise=true , address=45.91.116.11:4119 , piece_id=2FYRO5VRVNZX5FIL52RQOOQMAVVBN46V6J7IG5BPCG2KKLE6L3EA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.858614019 (886h56m12.729697065s) [noise=true , address=45.91.116.8:4046 , piece_id=RYR3PFTB2T66QYDW5PUBC4R7UQGT3SWLFOFQDM4CL4FIJVFOFOYA, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.881912943 (886h56m12.752995978s) [noise=true , address=145.79.227.27:4691 , piece_id=3BIVPAEULWSAIY6OL3UQGDJIBXC6DCLYMJGPI3R6JOUA4IS2H6FQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.898207971 (886h56m12.769291016s) [noise=true , address=145.79.227.20:4439 , piece_id=DBT5642MCGQ4RYEXRNUYHEOIRSO5GZMAOJ424QHLKG7JC5PXKQBQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:12.901932794 (886h56m12.773015829s) [noise=true , address=145.79.227.8:4330 , piece_id=AIFEBCOSC62KJRJUUTCZJSPKSLMIO7BEMETC4RBRLIYN5UXY25EQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.438979651 (886h56m13.310062696s) [noise=true , address=145.79.227.29:4941 , piece_id=LEOBO5XHIU6NLHXPV4CZJ3QUPTNSAPU2USLPZV6CD2KU7AINRTNQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611300972 (886h56m13.482384007s) [noise=true , address=145.79.227.44:4193 , piece_id=BT5DKCAZ65X5TJOM3NOR3VX6EXGRSKWBJU6FFUBHRLLWU3ASWVAQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: <nil> Jan 11 14:25:13.611335227 (886h56m13.482418273s) [seg_pos={0 0}, upload=776150]: Segment reached optimal threshold of 13 pieces. Jan 11 14:25:13.611561880 (886h56m13.482644925s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611568522 (886h56m13.482651557s) [noise=true , address=145.79.227.2:4113 , piece_id=XVOIV7SYOXVLSNXLWT36QMOYTIT5EACSMVJABQ7SSNMVEYE7LZMQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611740479 (886h56m13.482823515s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611748995 (886h56m13.48283203s) [noise=true , address=77.90.31.19:4844 , piece_id=NM6EWKLKLNSZFFI6H5DHTGRENYCFZV5ISHZYSSBNSYETTCCTH6CQ, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611758674 (886h56m13.482841719s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 11 14:25:13.611763484 (886h56m13.482846529s) [noise=true , address=45.91.116.4:4939 , piece_id=B2E7FNNCB22RI6CCRG3U4RMXRSRRFUTHRYH4SFNT6DZBRO7BD22A, seg_pos={0 0}, upload=776150]: Long tail cancellation due to optimal threshold Jan 11 14:25:13.611778903 (886h56m13.482861948s) [seg_pos={0 0}, upload=776150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 14:25:13.611858123 (886h56m13.482941168s) [upload=776150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:25:13.611862962 (886h56m13.482945997s) [upload=776150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:25:13.936800715 (886h56m13.80788376s) [upload=776150]: Done uploading segments... Jan 11 14:25:13.936821976 (886h56m13.807905051s) [upload=776150]: Upload finished. err: <nil> Duration: 5.730413496s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.96917826s trace: https://jaeger.dev.storj.io/trace/2c34a25ab27fc763 FAIL (2026-01-11 14:35:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.969215752s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [.X..XXXXXXXXXXXXXXXXXXX..XXXXXXXXXXX.X.XX.XXX.X.XXXXX..XXX.X] duration: 30.000884614s trace: https://jaeger.dev.storj.io/trace/1be2b04b0a40aa8d FAIL (2026-01-11 14:35:12 UTC) orbiter warning: context deadline exceeded Duration: 30.000906895s edge gateway us-select-1

1
...
389
390
391
392
393
...
9035