Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [X..X...XXXXXX.......X....X....X......XX......X......X..X....] duration:
3.56352418s trace:
https://jaeger.dev.storj.io/trace/52698545ec000935
FAIL (2025-12-25 14:44:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.563553696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.780924529s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:43:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.780955199s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
6.185160216s trace:
https://jaeger.dev.storj.io/trace/2464b09252149d1d
FAIL (2025-12-25 14:43:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q) context deadline exceeded
======LOGS======
Dec 25 14:41:53.719625840 (479h12m53.590708886s) [upload=420648]: Starting upload
Dec 25 14:41:53.719714038 (479h12m53.590797093s) [upload=420648]: Done starting upload
Dec 25 14:41:53.719815611 (479h12m53.590898656s) [upload=420648]: Uploading segments...
Dec 25 14:41:53.719987589 (479h12m53.591070624s) [upload=420648]: Got next segment. Inline: false
Dec 25 14:41:53.720028667 (479h12m53.591111702s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:41:53.720034468 (479h12m53.591117503s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:41:53.974566669 (479h12m53.845649724s) [seg_pos={0 0}, upload=420648]: Begin upload segment...
Dec 25 14:41:53.974622285 (479h12m53.845705331s) [seg_pos={0 0}, upload=420648]: Done begin upload segment.
Dec 25 14:41:53.974656471 (479h12m53.845739516s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.974706936 (479h12m53.845789971s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975246945 (479h12m53.84632999s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975622080 (479h12m53.846705125s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976017071 (479h12m53.847100106s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976412263 (479h12m53.847495329s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976803339 (479h12m53.847886384s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.977290787 (479h12m53.848373822s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978431939 (479h12m53.849514984s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978544464 (479h12m53.849627509s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.979619391 (479h12m53.850702426s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.980847840 (479h12m53.851930875s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.983019956 (479h12m53.854102991s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.984892983 (479h12m53.855976018s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.986444197 (479h12m53.857527242s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.988382928 (479h12m53.859465973s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:55.656144797 (479h12m55.527227842s) [upload=420648]: Next returned nil segment
Dec 25 14:41:55.656159716 (479h12m55.527242761s) [upload=420648]: Waiting for error group managing segments...
Dec 25 14:41:55.980587370 (479h12m55.851670405s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.980623830 (479h12m55.851706875s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.994117394 (479h12m55.865200439s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.001610682 (479h12m55.872693738s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004085315 (479h12m55.87516837s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004143506 (479h12m55.875226571s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004725414 (479h12m55.875808469s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.005029713 (479h12m55.876112778s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.009615144 (479h12m55.88069819s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.018810834 (479h12m55.88989388s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.302829424 (479h12m56.173912459s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.335735653 (479h12m56.206818698s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668297637 (479h12m56.539380682s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668326472 (479h12m56.539409517s) [seg_pos={0 0}, upload=420648]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:41:56.668678312 (479h12m56.539761377s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.668689994 (479h12m56.539773039s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669182391 (479h12m56.540265436s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669212619 (479h12m56.540295664s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669456283 (479h12m56.540539328s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669487041 (479h12m56.540570077s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669512500 (479h12m56.540595535s) [seg_pos={0 0}, upload=420648]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:41:56.669628441 (479h12m56.540711486s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:41:56.669655252 (479h12m56.540738297s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:41:56.965564873 (479h12m56.836647928s) [upload=420648]: Done uploading segments...
Dec 25 14:41:56.965590703 (479h12m56.836673748s) [upload=420648]: Upload finished. err: <nil>
Duration: 6.185181687s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.X..X...XXXXXX.......X....X....X......XX......X......X..X...] duration:
3.418853253s trace:
https://jaeger.dev.storj.io/trace/52698545ec000935
FAIL (2025-12-25 14:43:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.41887793s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
6.891137774s trace:
https://jaeger.dev.storj.io/trace/2464b09252149d1d
FAIL (2025-12-25 14:43:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q) context deadline exceeded
======LOGS======
Dec 25 14:41:53.719625840 (479h12m53.590708886s) [upload=420648]: Starting upload
Dec 25 14:41:53.719714038 (479h12m53.590797093s) [upload=420648]: Done starting upload
Dec 25 14:41:53.719815611 (479h12m53.590898656s) [upload=420648]: Uploading segments...
Dec 25 14:41:53.719987589 (479h12m53.591070624s) [upload=420648]: Got next segment. Inline: false
Dec 25 14:41:53.720028667 (479h12m53.591111702s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:41:53.720034468 (479h12m53.591117503s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:41:53.974566669 (479h12m53.845649724s) [seg_pos={0 0}, upload=420648]: Begin upload segment...
Dec 25 14:41:53.974622285 (479h12m53.845705331s) [seg_pos={0 0}, upload=420648]: Done begin upload segment.
Dec 25 14:41:53.974656471 (479h12m53.845739516s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.974706936 (479h12m53.845789971s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975246945 (479h12m53.84632999s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975622080 (479h12m53.846705125s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976017071 (479h12m53.847100106s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976412263 (479h12m53.847495329s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976803339 (479h12m53.847886384s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.977290787 (479h12m53.848373822s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978431939 (479h12m53.849514984s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978544464 (479h12m53.849627509s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.979619391 (479h12m53.850702426s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.980847840 (479h12m53.851930875s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.983019956 (479h12m53.854102991s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.984892983 (479h12m53.855976018s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.986444197 (479h12m53.857527242s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.988382928 (479h12m53.859465973s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:55.656144797 (479h12m55.527227842s) [upload=420648]: Next returned nil segment
Dec 25 14:41:55.656159716 (479h12m55.527242761s) [upload=420648]: Waiting for error group managing segments...
Dec 25 14:41:55.980587370 (479h12m55.851670405s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.980623830 (479h12m55.851706875s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.994117394 (479h12m55.865200439s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.001610682 (479h12m55.872693738s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004085315 (479h12m55.87516837s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004143506 (479h12m55.875226571s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004725414 (479h12m55.875808469s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.005029713 (479h12m55.876112778s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.009615144 (479h12m55.88069819s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.018810834 (479h12m55.88989388s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.302829424 (479h12m56.173912459s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.335735653 (479h12m56.206818698s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668297637 (479h12m56.539380682s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668326472 (479h12m56.539409517s) [seg_pos={0 0}, upload=420648]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:41:56.668678312 (479h12m56.539761377s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.668689994 (479h12m56.539773039s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669182391 (479h12m56.540265436s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669212619 (479h12m56.540295664s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669456283 (479h12m56.540539328s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669487041 (479h12m56.540570077s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669512500 (479h12m56.540595535s) [seg_pos={0 0}, upload=420648]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:41:56.669628441 (479h12m56.540711486s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:41:56.669655252 (479h12m56.540738297s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:41:56.965564873 (479h12m56.836647928s) [upload=420648]: Done uploading segments...
Dec 25 14:41:56.965590703 (479h12m56.836673748s) [upload=420648]: Upload finished. err: <nil>
Duration: 6.891167332s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
9.088578834s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:42:49 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.088596407s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [..X..X...XXXXXX.......X....X....X......XX......X......X..X..] duration:
4.190654532s trace:
https://jaeger.dev.storj.io/trace/52698545ec000935
FAIL (2025-12-25 14:42:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.190680081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.002651343s trace:
https://jaeger.dev.storj.io/trace/2464b09252149d1d
FAIL (2025-12-25 14:42:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q) context deadline exceeded
======LOGS======
Dec 25 14:41:53.719625840 (479h12m53.590708886s) [upload=420648]: Starting upload
Dec 25 14:41:53.719714038 (479h12m53.590797093s) [upload=420648]: Done starting upload
Dec 25 14:41:53.719815611 (479h12m53.590898656s) [upload=420648]: Uploading segments...
Dec 25 14:41:53.719987589 (479h12m53.591070624s) [upload=420648]: Got next segment. Inline: false
Dec 25 14:41:53.720028667 (479h12m53.591111702s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:41:53.720034468 (479h12m53.591117503s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:41:53.974566669 (479h12m53.845649724s) [seg_pos={0 0}, upload=420648]: Begin upload segment...
Dec 25 14:41:53.974622285 (479h12m53.845705331s) [seg_pos={0 0}, upload=420648]: Done begin upload segment.
Dec 25 14:41:53.974656471 (479h12m53.845739516s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.974706936 (479h12m53.845789971s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975246945 (479h12m53.84632999s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.975622080 (479h12m53.846705125s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976017071 (479h12m53.847100106s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976412263 (479h12m53.847495329s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.976803339 (479h12m53.847886384s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.977290787 (479h12m53.848373822s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978431939 (479h12m53.849514984s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.978544464 (479h12m53.849627509s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.979619391 (479h12m53.850702426s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.980847840 (479h12m53.851930875s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.983019956 (479h12m53.854102991s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.984892983 (479h12m53.855976018s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.986444197 (479h12m53.857527242s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:53.988382928 (479h12m53.859465973s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Uploading piece...
Dec 25 14:41:55.656144797 (479h12m55.527227842s) [upload=420648]: Next returned nil segment
Dec 25 14:41:55.656159716 (479h12m55.527242761s) [upload=420648]: Waiting for error group managing segments...
Dec 25 14:41:55.980587370 (479h12m55.851670405s) [noise=true , address=145.79.227.13:4802 , piece_id=ILEQKN33E7B3J2XKVFAUG5WGJEEB5ON63KL5IKYG63GSZV6BDOWA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.980623830 (479h12m55.851706875s) [noise=true , address=145.79.227.3:4149 , piece_id=F44ZHK6VEBKCFMF7YB5UEMCQAXIDVCRHNBMIQHHZPTC7INY574RA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:55.994117394 (479h12m55.865200439s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBXKQSOBYUCXRDRTV3DF24CPC5NSU6GWM32JO2RQXCYLTCQN3WBA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.001610682 (479h12m55.872693738s) [noise=true , address=145.79.227.45:4231 , piece_id=YIESEOIYY4Q7S5ENJQWSRYRPVCV5FGQRDMBUILX6YK2OS3NXVS5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004085315 (479h12m55.87516837s) [noise=true , address=145.79.227.29:4943 , piece_id=R6ASHKZY6V3OERFGZ6ZAROQA4BG5VD5J3YLIVBNTDELI4SM7FVQA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004143506 (479h12m55.875226571s) [noise=true , address=145.79.227.52:4447 , piece_id=RHZAOUKYC4QAKICT2ZSEIGZIARL34CPRGV4NWG6LDRPOFXUTNRRA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.004725414 (479h12m55.875808469s) [noise=true , address=145.79.227.50:4373 , piece_id=3K2NTODUNE6PYOHO45LOONWHTXCHFJUKT6YLHBG3QWTUG6CBKNRQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.005029713 (479h12m55.876112778s) [noise=true , address=145.79.227.41:4087 , piece_id=ISXUYPLSDSF3XB6F5IKF3F2VCCWNAJXFLSDX5GXUMVONHB7DBV7Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.009615144 (479h12m55.88069819s) [noise=true , address=145.79.227.14:4838 , piece_id=RJZBX327RFDUNC3QRAP5PZH3V4KOFZMZAQTXB56LZL7D4RQJTXEQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.018810834 (479h12m55.88989388s) [noise=true , address=145.79.227.35:4482 , piece_id=ZF5MJ4G577NOYRI7KSIACJRMP3GPLMY7COAIUUO7HLRSBYUWJWOA, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.302829424 (479h12m56.173912459s) [noise=true , address=45.91.116.9:4263 , piece_id=SPH5AHYRGLHCOHYL56NGQNGMJFKJCI44CFO5HHWJKTHDIDGDRP4A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.335735653 (479h12m56.206818698s) [noise=true , address=45.91.116.7:4010 , piece_id=DGYTIFF52HACE4S7TPWHCNPXWV3PTUAOAOJ4HHMX3I53GG5QBI2Q, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668297637 (479h12m56.539380682s) [noise=true , address=45.91.116.12:4228 , piece_id=WJGWJJFDBJKPOZAOL6MBN25VKORHDCQPMG5LMD3QGEKAEX275BPQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: <nil>
Dec 25 14:41:56.668326472 (479h12m56.539409517s) [seg_pos={0 0}, upload=420648]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:41:56.668678312 (479h12m56.539761377s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.668689994 (479h12m56.539773039s) [noise=true , address=145.79.227.40:4049 , piece_id=L762QG5UUZHAZXMW7YZG2U2ZYJ4EFCGKSAVQ44QOVIJRF3L44A5A, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669182391 (479h12m56.540265436s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669212619 (479h12m56.540295664s) [noise=true , address=77.90.31.14:4663 , piece_id=FVQIAN3WNMSEZRP3LI7HAWLZTGQS7PTZ7UVSLI23W27BNFIKKDNQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669456283 (479h12m56.540539328s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:56.669487041 (479h12m56.540570077s) [noise=true , address=45.91.116.5:4300 , piece_id=POWHLWTFMKYQGLFEPDAVJQRLRAWJZ5JBCPAEPNPQQVLOOSGNGQOQ, seg_pos={0 0}, upload=420648]: Long tail cancellation due to optimal threshold
Dec 25 14:41:56.669512500 (479h12m56.540595535s) [seg_pos={0 0}, upload=420648]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:41:56.669628441 (479h12m56.540711486s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:41:56.669655252 (479h12m56.540738297s) [upload=420648]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:41:56.965564873 (479h12m56.836647928s) [upload=420648]: Done uploading segments...
Dec 25 14:41:56.965590703 (479h12m56.836673748s) [upload=420648]: Upload finished. err: <nil>
Duration: 30.002664568s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
7.11449777s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:42:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.114619862s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.002616513s trace:
https://jaeger.dev.storj.io/trace/617ac916c565e4c4
FAIL (2025-12-25 14:41:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: STJT364DQVJF33SQW3HCQPQBG77XQMFAXXOWBZQA3QQL326NMBEA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: D77CGW3Z5P77RCCLDOU4B44L7HOR5MLF6LSRZB2DAF6N4EU5GUSQ) context deadline exceeded
======LOGS======
c 25 14:41:23.485947225 (479h12m23.35703027s) [upload=420642]: Starting upload
Dec 25 14:41:23.486019764 (479h12m23.357102809s) [upload=420642]: Done starting upload
Dec 25 14:41:23.486114233 (479h12m23.357197268s) [upload=420642]: Uploading segments...
Dec 25 14:41:23.486270270 (479h12m23.357353315s) [upload=420642]: Got next segment. Inline: false
Dec 25 14:41:23.486319293 (479h12m23.357402338s) [upload=420642]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 25 14:41:23.486324934 (479h12m23.357407969s) [upload=420642]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 25 14:41:23.731025857 (479h12m23.602108912s) [seg_pos={0 0}, upload=420642]: Begin upload segment...
Dec 25 14:41:23.731085951 (479h12m23.602168996s) [seg_pos={0 0}, upload=420642]: Done begin upload segment.
Dec 25 14:41:23.731112662 (479h12m23.602195697s) [noise=true , address=145.79.227.32:4835 , piece_id=SAINVSWZSJM3IHF2NGAQG7BP74RCF3PNWHKEEFBOEHAXXBDEKFCA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.731121619 (479h12m23.602204664s) [noise=true , address=145.79.227.52:4446 , piece_id=4LT2ELNPFTHV5SIBTZDWUN3CHSZXENFDAKB2C6JQJM3KAGUWFYEA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.731692887 (479h12m23.602775942s) [noise=true , address=145.79.227.19:4077 , piece_id=XYWOYI5OQRSTS3TDYO6TEQVR76XNHILE6FWBGXVJKZS4HFHU6DEA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.732206104 (479h12m23.603289149s) [noise=true , address=145.79.227.9:4367 , piece_id=MVXXFEV43TW2HMMG2OMLM66MYBCSSSCOVMNHQTF7K5KJCT7S6QKA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.732594434 (479h12m23.603677469s) [noise=true , address=45.91.116.8:4045 , piece_id=STJT364DQVJF33SQW3HCQPQBG77XQMFAXXOWBZQA3QQL326NMBEA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.732992401 (479h12m23.604075446s) [noise=true , address=145.79.227.14:4839 , piece_id=JF52FLYC73KZFZTPM5VQTUUJMVSDHBZIMVEKCNZUMLC6UGFBBGGQ, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.733498585 (479h12m23.60458162s) [noise=true , address=145.79.227.24:4583 , piece_id=QNGONQKORPKYJUOR3JYG4KZKLSHTYRDXHP55GYFYR6HI3AJW232A, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.733963410 (479h12m23.605046446s) [noise=true , address=77.90.31.14:4661 , piece_id=AKW4NJLN5PFLFYCOK7YUUPR57R4F33C42L6D3BR7HBEAVTXQ5GLQ, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.734527394 (479h12m23.605610429s) [noise=true , address=145.79.227.51:4409 , piece_id=ANV5UEFVZZK3WRTSOFP3EUIXDX4UGHE433HBXGIFU2MQK4NPDNWA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.735651304 (479h12m23.606734339s) [noise=true , address=77.90.31.13:4629 , piece_id=4M4BAKWWKH7U2AZCQPSZRMHCRWKC6ECHV3OAAJF6AADDL6PLFIEA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.736085751 (479h12m23.607168806s) [noise=true , address=145.79.227.30:4762 , piece_id=F32XUPSPWMD2CCZEDJ6LWAOPRF32OGWUOAUFPUXY62NOWQFCMZDA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.736991265 (479h12m23.60807431s) [noise=true , address=145.79.227.48:4303 , piece_id=I7AIOLJ4OP6LUETX3ZGGFLVOPJF5WEJ3VN2CDOQ2QT4IRTL5MODA, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.737860410 (479h12m23.608943455s) [noise=true , address=145.79.227.50:4373 , piece_id=7BT4X6C3LORXYIBW2UBM7IBMWZTIEQQ3MKL5HOABOD4JEVHVEJ4A, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.738635867 (479h12m23.609718912s) [noise=true , address=145.79.227.6:4259 , piece_id=WTPFP7H3GBBIWELH4N3OTQQJV4YIFRXJ5BILVPTVJSDRHSSUMJPQ, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.739699472 (479h12m23.610782507s) [noise=true , address=45.91.116.7:4011 , piece_id=D77CGW3Z5P77RCCLDOU4B44L7HOR5MLF6LSRZB2DAF6N4EU5GUSQ, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:23.740416378 (479h12m23.611499413s) [noise=true , address=145.79.227.36:4519 , piece_id=5ZERAVOLCWFP2UAIM673KYGIPAIMKYRXOH6SVQCEBIS4SUL2FDFQ, seg_pos={0 0}, upload=420642]: Uploading piece...
Dec 25 14:41:25.458234642 (479h12m25.329317687s) [upload=420642]: Next returned nil segment
Dec 25 14:41:25.458307210 (479h12m25.329390255s) [upload=420642]: Waiting for error group managing segments...
Dec 25 14:41:25.745618829 (479h12m25.616701874s) [noise=true , address=145.79.227.51:4409 , piece_id=ANV5UEFVZZK3WRTSOFP3EUIXDX4UGHE433HBXGIFU2MQK4NPDNWA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.746672456 (479h12m25.617755491s) [noise=true , address=145.79.227.36:4519 , piece_id=5ZERAVOLCWFP2UAIM673KYGIPAIMKYRXOH6SVQCEBIS4SUL2FDFQ, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.751046435 (479h12m25.62212947s) [noise=true , address=145.79.227.32:4835 , piece_id=SAINVSWZSJM3IHF2NGAQG7BP74RCF3PNWHKEEFBOEHAXXBDEKFCA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.755606598 (479h12m25.626689633s) [noise=true , address=145.79.227.24:4583 , piece_id=QNGONQKORPKYJUOR3JYG4KZKLSHTYRDXHP55GYFYR6HI3AJW232A, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.756269380 (479h12m25.627352415s) [noise=true , address=145.79.227.19:4077 , piece_id=XYWOYI5OQRSTS3TDYO6TEQVR76XNHILE6FWBGXVJKZS4HFHU6DEA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.762913832 (479h12m25.633996877s) [noise=true , address=145.79.227.9:4367 , piece_id=MVXXFEV43TW2HMMG2OMLM66MYBCSSSCOVMNHQTF7K5KJCT7S6QKA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.766585594 (479h12m25.637668629s) [noise=true , address=145.79.227.52:4446 , piece_id=4LT2ELNPFTHV5SIBTZDWUN3CHSZXENFDAKB2C6JQJM3KAGUWFYEA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.767514853 (479h12m25.638597889s) [noise=true , address=145.79.227.50:4373 , piece_id=7BT4X6C3LORXYIBW2UBM7IBMWZTIEQQ3MKL5HOABOD4JEVHVEJ4A, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.773789592 (479h12m25.644872637s) [noise=true , address=145.79.227.48:4303 , piece_id=I7AIOLJ4OP6LUETX3ZGGFLVOPJF5WEJ3VN2CDOQ2QT4IRTL5MODA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.778608257 (479h12m25.649691302s) [noise=true , address=145.79.227.14:4839 , piece_id=JF52FLYC73KZFZTPM5VQTUUJMVSDHBZIMVEKCNZUMLC6UGFBBGGQ, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:25.999934349 (479h12m25.871017384s) [noise=true , address=77.90.31.13:4629 , piece_id=4M4BAKWWKH7U2AZCQPSZRMHCRWKC6ECHV3OAAJF6AADDL6PLFIEA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:26.114549396 (479h12m25.985632441s) [noise=true , address=45.91.116.8:4045 , piece_id=STJT364DQVJF33SQW3HCQPQBG77XQMFAXXOWBZQA3QQL326NMBEA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:26.117759269 (479h12m25.988842314s) [noise=true , address=45.91.116.7:4011 , piece_id=D77CGW3Z5P77RCCLDOU4B44L7HOR5MLF6LSRZB2DAF6N4EU5GUSQ, seg_pos={0 0}, upload=420642]: Done uploading piece. err: <nil>
Dec 25 14:41:26.117782422 (479h12m25.988865467s) [seg_pos={0 0}, upload=420642]: Segment reached optimal threshold of 13 pieces.
Dec 25 14:41:26.118054911 (479h12m25.989137956s) [noise=true , address=77.90.31.14:4661 , piece_id=AKW4NJLN5PFLFYCOK7YUUPR57R4F33C42L6D3BR7HBEAVTXQ5GLQ, seg_pos={0 0}, upload=420642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:46506->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:46506->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46506->77.90.31.14:4661: use of closed network connection
Dec 25 14:41:26.118064008 (479h12m25.989147073s) [noise=true , address=77.90.31.14:4661 , piece_id=AKW4NJLN5PFLFYCOK7YUUPR57R4F33C42L6D3BR7HBEAVTXQ5GLQ, seg_pos={0 0}, upload=420642]: Long tail cancellation due to optimal threshold
Dec 25 14:41:26.118345544 (479h12m25.989428579s) [noise=true , address=145.79.227.6:4259 , piece_id=WTPFP7H3GBBIWELH4N3OTQQJV4YIFRXJ5BILVPTVJSDRHSSUMJPQ, seg_pos={0 0}, upload=420642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:26.118388907 (479h12m25.989471942s) [noise=true , address=145.79.227.6:4259 , piece_id=WTPFP7H3GBBIWELH4N3OTQQJV4YIFRXJ5BILVPTVJSDRHSSUMJPQ, seg_pos={0 0}, upload=420642]: Long tail cancellation due to optimal threshold
Dec 25 14:41:26.118468429 (479h12m25.989551484s) [noise=true , address=145.79.227.30:4762 , piece_id=F32XUPSPWMD2CCZEDJ6LWAOPRF32OGWUOAUFPUXY62NOWQFCMZDA, seg_pos={0 0}, upload=420642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 25 14:41:26.118478187 (479h12m25.989561222s) [noise=true , address=145.79.227.30:4762 , piece_id=F32XUPSPWMD2CCZEDJ6LWAOPRF32OGWUOAUFPUXY62NOWQFCMZDA, seg_pos={0 0}, upload=420642]: Long tail cancellation due to optimal threshold
Dec 25 14:41:26.118505740 (479h12m25.989588775s) [seg_pos={0 0}, upload=420642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 25 14:41:26.118592235 (479h12m25.98967528s) [upload=420642]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 25 14:41:26.118598947 (479h12m25.989682002s) [upload=420642]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 25 14:41:26.413879250 (479h12m26.284962285s) [upload=420642]: Done uploading segments...
Dec 25 14:41:26.413898286 (479h12m26.284981321s) [upload=420642]: Upload finished. err: <nil>
Duration: 30.002640449s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
10.385381002s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:41:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.385404385s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [X..X..X...XXXXXX.......X....X....X......XX......X......X..X.] duration:
5.006575651s trace:
https://jaeger.dev.storj.io/trace/52698545ec000935
FAIL (2025-12-25 14:41:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.006598676s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.547415104s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:41:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.547440392s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [XX..X..X...XXXXXX.......X....X....X......XX......X......X..X] duration:
5.001514262s trace:
https://jaeger.dev.storj.io/trace/52698545ec000935
FAIL (2025-12-25 14:41:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001542636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [XXX..X..X...XXXXXX.......X....X....X......XX......X......X..] duration:
5.001218967s trace:
https://jaeger.dev.storj.io/trace/33669c697a67975b
FAIL (2025-12-25 14:40:16 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001306025s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.700779593s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:40:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.700832585s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [XXXX..X..X...XXXXXX.......X....X....X......XX......X......X.] duration:
3.631856992s trace:
https://jaeger.dev.storj.io/trace/33669c697a67975b
FAIL (2025-12-25 14:39:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.631904863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
9.066680775s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:38:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.066758543s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [XXXXX..X..X...XXXXXX.......X....X....X......XX......X......X] duration:
5.001584042s trace:
https://jaeger.dev.storj.io/trace/33669c697a67975b
FAIL (2025-12-25 14:38:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001608428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [XXXXXX..X..X...XXXXXX.......X....X....X......XX......X......] duration:
3.598581911s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:38:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.598598303s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
8.111364563s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:37:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.111391074s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
11.648628756s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:37:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.648653472s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [.XXXXXX..X..X...XXXXXX.......X....X....X......XX......X.....] duration:
3.63254376s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:37:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.632571213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.............................X.............................] duration:
10.317348218s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:36:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.317376471s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [..XXXXXX..X..X...XXXXXX.......X....X....X......XX......X....] duration:
5.001836634s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:36:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00188133s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X.............................X............................] duration:
14.417211151s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:35:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.417241841s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [X..XXXXXX..X..X...XXXXXX.......X....X....X......XX......X...] duration:
4.560759599s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:35:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.56078649s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX......X..] duration:
4.942603483s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:35:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.94263319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X.............................X...........................] duration:
18.893165744s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:34:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.893235607s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX......X.] duration:
4.928110679s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:34:04 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.928172136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X.............................X..........................] duration:
10.380577957s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:33:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.380601981s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X.............................X.........................] duration:
14.501434147s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:33:05 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.501460808s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX......X] duration:
5.001661682s trace:
https://jaeger.dev.storj.io/trace/5772e4d11a1e93cb
FAIL (2025-12-25 14:32:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001683885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X.............................X........................] duration:
10.329311865s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:32:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.329341663s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X.............................X.......................] duration:
13.515319707s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:32:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.515373259s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX......] duration:
5.00049003s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:31:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000623844s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.............................X......................] duration:
17.113632767s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:31:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.113661142s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX.....] duration:
5.00122647s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:31:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001243282s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.............................X.....................] duration:
15.046190359s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:30:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.046213975s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X.............................X....................] duration:
13.541018577s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:29:58 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.541043645s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [.X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX....] duration:
4.289472109s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:29:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.289560738s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [X.X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX...] duration:
4.99525448s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:28:58 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.995283556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X.............................X...................] duration:
12.545224195s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:28:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.545251977s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X.............................X..................] duration:
7.909187524s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:28:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.909214335s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [XX.X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX..] duration:
4.926046475s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:27:58 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.926070271s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X.............................X.................] duration:
16.809913215s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:27:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.809940226s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [XXX.X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX.] duration:
3.60339698s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:26:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.603421566s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X.............................X................] duration:
10.938992682s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:26:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.93901246s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
27 out of 60 is failed (threshold=6) [XXXX.X.X.XX..XXXXXX..X..X...XXXXXX.......X....X....X......XX] duration:
5.001716766s trace:
https://jaeger.dev.storj.io/trace/13483525c301cd9
FAIL (2025-12-25 14:25:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001741875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X.............................X...............] duration:
15.242191306s trace:
https://jaeger.dev.storj.io/trace/4a034410a446d28d
FAIL (2025-12-25 14:25:26 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.242223848s
edge
gateway
us-select-1
|