Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X..........X.....X...X...X........XX] duration: 30.001735737s trace: https://jaeger.dev.storj.io/trace/1e79bf0cbb46f7f9 FAIL (2026-04-17 22:38:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ments... Apr 17 22:37:58.471270102 (246h51m45.290506126s) [upload=237005]: Got next segment. Inline: false Apr 17 22:37:58.471378241 (246h51m45.290614265s) [upload=237005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:37:58.471387018 (246h51m45.290623052s) [upload=237005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:37:58.708193142 (246h51m45.527429166s) [seg_pos={0 0}, upload=237005]: Begin upload segment... Apr 17 22:37:58.708247858 (246h51m45.527483882s) [seg_pos={0 0}, upload=237005]: Done begin upload segment. Apr 17 22:37:58.708272956 (246h51m45.52750898s) [noise=true , address=77.90.31.13:4630 , piece_id=IZY2WSNNJG5ROEOM6CBH6YD25ZRZIE7XMZMSQO6XQ7H2ESKHJ6UQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.708361497 (246h51m45.527597561s) [noise=true , address=145.79.227.6:4262 , piece_id=BUVADRII3ULSE4HZZHPBAWDV5D2CXIL7WZAJ4DYVZP6QCLHADVDQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.708820833 (246h51m45.528056857s) [noise=true , address=45.91.116.6:4989 , piece_id=L5BSUBS32DI74H3T2I56EM3EPCXI7SJ2KFCDHQOYZ76J5NDP5MBA, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.708977306 (246h51m45.52821333s) [noise=true , address=77.90.31.15:4700 , piece_id=Q4YKXK7S5UXV3VGQDXC7RLVWAQVEJ3BS2WUY3LGG2KXFU4PTTXFA, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709119320 (246h51m45.528355344s) [noise=true , address=145.79.227.39:4016 , piece_id=CGNLPBWTFLFB5K3ADI5BX75J7ERJECZQPVAF7UTC622WYW3JD23A, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709308925 (246h51m45.52854495s) [noise=true , address=45.91.116.8:4048 , piece_id=WBZ7RKVOR3RDRAWHOM6RACJGR4ZQMWBOSBAYMZ4DPNNSZTWM7GPA, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709392147 (246h51m45.528628171s) [noise=true , address=145.79.227.45:4233 , piece_id=CCSYLQB2SH2O5BUVROK43GOOD2XSN3BDYT72SWBA4GTKLAU2GDQA, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709605307 (246h51m45.528841331s) [noise=true , address=145.79.227.52:4449 , piece_id=AC4DG67AWU5GNPBOKPHPPIXVZWXZWQJBSWJTK6RRWSKJ7EJWCQKQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709666606 (246h51m45.52890263s) [noise=true , address=145.79.227.19:4078 , piece_id=O4QR7JF6XGDT5HWLQGSBUQL3RBT54VSJSIDRJWBLCVRJEXOWEOEQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709922579 (246h51m45.529158603s) [noise=true , address=145.79.227.21:4473 , piece_id=YEYRKNQ2D2466MWD5TOASDZO6NVLXTQZ7HEHMT4JPBTFQO3Q6V2Q, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.709971264 (246h51m45.529207288s) [noise=true , address=145.79.227.5:4225 , piece_id=55MLEQTSPA5UWDPJAWWWV6AZ5GACSPSY7KORDN2YUPVG2FAMQIKQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.710303314 (246h51m45.529539338s) [noise=true , address=77.90.31.14:4666 , piece_id=HDSXNC6ZCVZZDTHDBUZGPFXJQT76Z6FMZROKJ4ZGHD5MUUSFIW3A, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.710310148 (246h51m45.529546172s) [noise=true , address=145.79.227.25:4623 , piece_id=ZTR3BUXES2W5NVFPIT4RXIN2WOIHJUWLL6KFYBPWQKTOCKY7RZOQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.710527357 (246h51m45.529763381s) [noise=true , address=77.90.31.10:4521 , piece_id=2USPTFNSVT4QD3MNVHFPDE5GZW3KXARF3SRW3XLOSJXA6RANY4VA, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.710597883 (246h51m45.529833907s) [noise=true , address=45.91.116.13:4188 , piece_id=RTVKZTFBJAI5LWLP3G5ST5AHOWLCJME3MZGTZG4BWU3PE2T2WE5A, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:37:58.710890227 (246h51m45.530126251s) [noise=true , address=45.91.116.14:4224 , piece_id=ZSZKRRRCUEV36AXYRUSEN2YHCUMVH4D7N4KVEAWQ5OF4XEWZD3JQ, seg_pos={0 0}, upload=237005]: Uploading piece... Apr 17 22:38:00.506886217 (246h51m47.326122241s) [upload=237005]: Next returned nil segment Apr 17 22:38:00.506910955 (246h51m47.326146989s) [upload=237005]: Waiting for error group managing segments... Apr 17 22:38:00.795192183 (246h51m47.614428217s) [noise=true , address=77.90.31.14:4666 , piece_id=HDSXNC6ZCVZZDTHDBUZGPFXJQT76Z6FMZROKJ4ZGHD5MUUSFIW3A, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:00.802966977 (246h51m47.622203012s) [noise=true , address=77.90.31.10:4521 , piece_id=2USPTFNSVT4QD3MNVHFPDE5GZW3KXARF3SRW3XLOSJXA6RANY4VA, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:00.837150420 (246h51m47.656386444s) [noise=true , address=77.90.31.15:4700 , piece_id=Q4YKXK7S5UXV3VGQDXC7RLVWAQVEJ3BS2WUY3LGG2KXFU4PTTXFA, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:00.857273577 (246h51m47.676509601s) [noise=true , address=77.90.31.13:4630 , piece_id=IZY2WSNNJG5ROEOM6CBH6YD25ZRZIE7XMZMSQO6XQ7H2ESKHJ6UQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:01.831163825 (246h51m48.650399849s) [noise=true , address=45.91.116.13:4188 , piece_id=RTVKZTFBJAI5LWLP3G5ST5AHOWLCJME3MZGTZG4BWU3PE2T2WE5A, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.129931912 (246h51m48.949167936s) [noise=true , address=145.79.227.39:4016 , piece_id=CGNLPBWTFLFB5K3ADI5BX75J7ERJECZQPVAF7UTC622WYW3JD23A, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.167981246 (246h51m48.98721728s) [noise=true , address=45.91.116.14:4224 , piece_id=ZSZKRRRCUEV36AXYRUSEN2YHCUMVH4D7N4KVEAWQ5OF4XEWZD3JQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.179145783 (246h51m48.998381807s) [noise=true , address=45.91.116.8:4048 , piece_id=WBZ7RKVOR3RDRAWHOM6RACJGR4ZQMWBOSBAYMZ4DPNNSZTWM7GPA, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.275673416 (246h51m49.09490945s) [noise=true , address=145.79.227.52:4449 , piece_id=AC4DG67AWU5GNPBOKPHPPIXVZWXZWQJBSWJTK6RRWSKJ7EJWCQKQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.654733063 (246h51m49.473969097s) [noise=true , address=145.79.227.6:4262 , piece_id=BUVADRII3ULSE4HZZHPBAWDV5D2CXIL7WZAJ4DYVZP6QCLHADVDQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.733948735 (246h51m49.553184759s) [noise=true , address=145.79.227.25:4623 , piece_id=ZTR3BUXES2W5NVFPIT4RXIN2WOIHJUWLL6KFYBPWQKTOCKY7RZOQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.806700572 (246h51m49.625936596s) [noise=true , address=145.79.227.21:4473 , piece_id=YEYRKNQ2D2466MWD5TOASDZO6NVLXTQZ7HEHMT4JPBTFQO3Q6V2Q, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.807972947 (246h51m49.627208972s) [noise=true , address=145.79.227.45:4233 , piece_id=CCSYLQB2SH2O5BUVROK43GOOD2XSN3BDYT72SWBA4GTKLAU2GDQA, seg_pos={0 0}, upload=237005]: Done uploading piece. err: <nil> Apr 17 22:38:02.807993647 (246h51m49.627229681s) [seg_pos={0 0}, upload=237005]: Satisfied cohort requirements after 13 pieces Apr 17 22:38:02.808282966 (246h51m49.62751901s) [noise=true , address=45.91.116.6:4989 , piece_id=L5BSUBS32DI74H3T2I56EM3EPCXI7SJ2KFCDHQOYZ76J5NDP5MBA, seg_pos={0 0}, upload=237005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs): protocol: expected piece hash; context canceled; EOF Apr 17 22:38:02.808293136 (246h51m49.62752917s) [noise=true , address=45.91.116.6:4989 , piece_id=L5BSUBS32DI74H3T2I56EM3EPCXI7SJ2KFCDHQOYZ76J5NDP5MBA, seg_pos={0 0}, upload=237005]: Long tail cancellation due to optimal threshold Apr 17 22:38:02.808429940 (246h51m49.627665964s) [noise=true , address=145.79.227.19:4078 , piece_id=O4QR7JF6XGDT5HWLQGSBUQL3RBT54VSJSIDRJWBLCVRJEXOWEOEQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:41300->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:41300->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41300->145.79.227.19:4078: use of closed network connection Apr 17 22:38:02.808441132 (246h51m49.627677166s) [noise=true , address=145.79.227.19:4078 , piece_id=O4QR7JF6XGDT5HWLQGSBUQL3RBT54VSJSIDRJWBLCVRJEXOWEOEQ, seg_pos={0 0}, upload=237005]: Long tail cancellation due to optimal threshold Apr 17 22:38:02.808538750 (246h51m49.627774784s) [noise=true , address=145.79.227.5:4225 , piece_id=55MLEQTSPA5UWDPJAWWWV6AZ5GACSPSY7KORDN2YUPVG2FAMQIKQ, seg_pos={0 0}, upload=237005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY): protocol: write tcp 188.166.225.170:44240->145.79.227.5:4225: use of closed network connection; write tcp 188.166.225.170:44240->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44240->145.79.227.5:4225: use of closed network connection Apr 17 22:38:02.808549581 (246h51m49.627785615s) [noise=true , address=145.79.227.5:4225 , piece_id=55MLEQTSPA5UWDPJAWWWV6AZ5GACSPSY7KORDN2YUPVG2FAMQIKQ, seg_pos={0 0}, upload=237005]: Long tail cancellation due to optimal threshold Apr 17 22:38:02.808579067 (246h51m49.627815091s) [seg_pos={0 0}, upload=237005]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:38:02.808667418 (246h51m49.627903452s) [upload=237005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:38:02.808674110 (246h51m49.627910134s) [upload=237005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:38:03.086673887 (246h51m49.905909911s) [upload=237005]: Done uploading segments... Apr 17 22:38:03.086701991 (246h51m49.905938015s) [upload=237005]: Upload finished. err: <nil> Duration: 30.001746928s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XXXX.XX] duration: 5.001333548s trace: https://jaeger.dev.storj.io/trace/1df1a2816f7b4e82 FAIL (2026-04-17 22:37:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0013556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..........X.....X...X...X........X] duration: 30.00258912s trace: https://jaeger.dev.storj.io/trace/7494308d9aacf4f1 FAIL (2026-04-17 22:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Apr 17 22:37:28.214244936 (246h51m15.03348096s) [upload=236994]: Got next segment. Inline: false Apr 17 22:37:28.214277869 (246h51m15.033513894s) [upload=236994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:37:28.214283811 (246h51m15.033519835s) [upload=236994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:37:28.458578086 (246h51m15.277814111s) [seg_pos={0 0}, upload=236994]: Begin upload segment... Apr 17 22:37:28.458640887 (246h51m15.277876921s) [seg_pos={0 0}, upload=236994]: Done begin upload segment. Apr 17 22:37:28.458647731 (246h51m15.277883755s) [noise=true , address=45.91.116.4:4953 , piece_id=O74AWJNRLHZ7PHX7RUWF24F525HLPBILZE42YOYGE56QUFWVWPFQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.458754507 (246h51m15.277990561s) [noise=true , address=145.79.227.31:4798 , piece_id=TK7OW452WQC4RYNM4N5LU42BQQTXDLIRWJOTS6LM63JKZMKQ223A, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.459244172 (246h51m15.278480196s) [noise=true , address=45.91.116.8:4048 , piece_id=S62KXMOD4HX4UKV5PA7J5E4TM3ZZXQIZIYMGXE5GEVGD3NZDN7SA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.459554320 (246h51m15.278790344s) [noise=true , address=145.79.227.19:4083 , piece_id=FDL7VEBDBFRAXNLOOWRWPNEYX256TBKEQEL2AGLF2UGBZYBLR5YA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.459801588 (246h51m15.279037612s) [noise=true , address=145.79.227.46:4603 , piece_id=DEPOFSAVLETIXFO6ZJASBFADJBCYMC5ESAQY552RGHJHO3IX6QSQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.460110715 (246h51m15.279346739s) [noise=true , address=145.79.227.14:4838 , piece_id=E4PH3HQPDHCJT3H4PWPL6ZOWZYN7V45O7KA2EF44E6EUATWDE5SA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.460463185 (246h51m15.279699209s) [noise=true , address=145.79.227.29:4942 , piece_id=7VZYDOOXGTTRXZGHOXAYDMNFJEY7OLD5AYQRKAOJMM2PFLTPHD4A, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.460722195 (246h51m15.279958219s) [noise=true , address=145.79.227.5:4227 , piece_id=4NMIZ2MIK4B3UBBEUKUO43AJ6TGYU4GJELTASVKIJI4ESZX4I6KA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.461011243 (246h51m15.280247267s) [noise=true , address=145.79.227.2:4114 , piece_id=WZVHVBXMEFWOXMIBNACQIJJVBK3ZMMQTD2OXSBCKYHFN6DMB3ZAQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.461998819 (246h51m15.281234843s) [noise=true , address=77.90.31.10:4519 , piece_id=CZSOA5WJ6RONOC3LMEH6P7BJMK6CAVUDPBZES5CSJAFDGAVTQUOQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.463005010 (246h51m15.282241034s) [noise=true , address=77.90.31.13:4625 , piece_id=6W4X25IOXSEQ5TDEFWCWGIG5UXRHSLOGJZACMZVBCHCWY6C5EE4A, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.463906270 (246h51m15.283142294s) [noise=true , address=145.79.227.26:4658 , piece_id=556C7CKSIQXWHA5QQCPZUHCR5HGKQLGBMVGMLFWX5V5A6ISGMHXQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.464891651 (246h51m15.284127675s) [noise=true , address=77.90.31.12:4594 , piece_id=HMQGNFRPXHMOHE77F6FUVC5LCYPWG2OSG3KNH67EGRXQXNCY7WRA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.465836415 (246h51m15.285072439s) [noise=true , address=145.79.227.50:4374 , piece_id=VBITVRFP6VD3ZVFKRVLYCKLMYNXEGDOHU2ZG4HCDKY6WMKLJ2GRA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.466764726 (246h51m15.28600075s) [noise=true , address=145.79.227.25:4617 , piece_id=22QDOICJRRBER5JTXUK2OIO6W6GK3PND3NGNXWQJXVCY3LOAECMQ, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:28.467743064 (246h51m15.286979089s) [noise=true , address=145.79.227.36:4521 , piece_id=OEYAZ7TB5X34T3IRMDZB77UGMZKPRQEFJUBJL2WXEDJ7EED55NXA, seg_pos={0 0}, upload=236994]: Uploading piece... Apr 17 22:37:30.278864435 (246h51m17.098100459s) [upload=236994]: Next returned nil segment Apr 17 22:37:30.278876839 (246h51m17.098112863s) [upload=236994]: Waiting for error group managing segments... Apr 17 22:37:30.566086349 (246h51m17.385322373s) [noise=true , address=77.90.31.10:4519 , piece_id=CZSOA5WJ6RONOC3LMEH6P7BJMK6CAVUDPBZES5CSJAFDGAVTQUOQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:30.570151815 (246h51m17.389387829s) [noise=true , address=77.90.31.12:4594 , piece_id=HMQGNFRPXHMOHE77F6FUVC5LCYPWG2OSG3KNH67EGRXQXNCY7WRA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:30.752385085 (246h51m17.571621119s) [noise=true , address=77.90.31.13:4625 , piece_id=6W4X25IOXSEQ5TDEFWCWGIG5UXRHSLOGJZACMZVBCHCWY6C5EE4A, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.497382272 (246h51m18.316618316s) [noise=true , address=145.79.227.46:4603 , piece_id=DEPOFSAVLETIXFO6ZJASBFADJBCYMC5ESAQY552RGHJHO3IX6QSQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.505716327 (246h51m18.324952351s) [noise=true , address=145.79.227.26:4658 , piece_id=556C7CKSIQXWHA5QQCPZUHCR5HGKQLGBMVGMLFWX5V5A6ISGMHXQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.551590903 (246h51m18.370826967s) [noise=true , address=145.79.227.36:4521 , piece_id=OEYAZ7TB5X34T3IRMDZB77UGMZKPRQEFJUBJL2WXEDJ7EED55NXA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.621855759 (246h51m18.441091783s) [noise=true , address=145.79.227.19:4083 , piece_id=FDL7VEBDBFRAXNLOOWRWPNEYX256TBKEQEL2AGLF2UGBZYBLR5YA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.805903783 (246h51m18.625139817s) [noise=true , address=145.79.227.31:4798 , piece_id=TK7OW452WQC4RYNM4N5LU42BQQTXDLIRWJOTS6LM63JKZMKQ223A, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:31.948573067 (246h51m18.767809091s) [noise=true , address=45.91.116.4:4953 , piece_id=O74AWJNRLHZ7PHX7RUWF24F525HLPBILZE42YOYGE56QUFWVWPFQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:32.229724636 (246h51m19.04896066s) [noise=true , address=145.79.227.2:4114 , piece_id=WZVHVBXMEFWOXMIBNACQIJJVBK3ZMMQTD2OXSBCKYHFN6DMB3ZAQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:32.315511009 (246h51m19.134747033s) [noise=true , address=145.79.227.50:4374 , piece_id=VBITVRFP6VD3ZVFKRVLYCKLMYNXEGDOHU2ZG4HCDKY6WMKLJ2GRA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:32.502938084 (246h51m19.322174108s) [noise=true , address=145.79.227.25:4617 , piece_id=22QDOICJRRBER5JTXUK2OIO6W6GK3PND3NGNXWQJXVCY3LOAECMQ, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:32.640789250 (246h51m19.460025274s) [noise=true , address=45.91.116.8:4048 , piece_id=S62KXMOD4HX4UKV5PA7J5E4TM3ZZXQIZIYMGXE5GEVGD3NZDN7SA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: <nil> Apr 17 22:37:32.640811472 (246h51m19.460047496s) [seg_pos={0 0}, upload=236994]: Satisfied cohort requirements after 13 pieces Apr 17 22:37:32.640989757 (246h51m19.460225781s) [noise=true , address=145.79.227.29:4942 , piece_id=7VZYDOOXGTTRXZGHOXAYDMNFJEY7OLD5AYQRKAOJMM2PFLTPHD4A, seg_pos={0 0}, upload=236994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Apr 17 22:37:32.640996579 (246h51m19.460232603s) [noise=true , address=145.79.227.29:4942 , piece_id=7VZYDOOXGTTRXZGHOXAYDMNFJEY7OLD5AYQRKAOJMM2PFLTPHD4A, seg_pos={0 0}, upload=236994]: Long tail cancellation due to optimal threshold Apr 17 22:37:32.641271821 (246h51m19.460507855s) [noise=true , address=145.79.227.5:4227 , piece_id=4NMIZ2MIK4B3UBBEUKUO43AJ6TGYU4GJELTASVKIJI4ESZX4I6KA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA): protocol: write tcp 188.166.225.170:44730->145.79.227.5:4227: use of closed network connection; write tcp 188.166.225.170:44730->145.79.227.5:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44730->145.79.227.5:4227: use of closed network connection Apr 17 22:37:32.641279396 (246h51m19.46051542s) [noise=true , address=145.79.227.5:4227 , piece_id=4NMIZ2MIK4B3UBBEUKUO43AJ6TGYU4GJELTASVKIJI4ESZX4I6KA, seg_pos={0 0}, upload=236994]: Long tail cancellation due to optimal threshold Apr 17 22:37:32.641378857 (246h51m19.460614881s) [noise=true , address=145.79.227.14:4838 , piece_id=E4PH3HQPDHCJT3H4PWPL6ZOWZYN7V45O7KA2EF44E6EUATWDE5SA, seg_pos={0 0}, upload=236994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:54480->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:54480->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54480->145.79.227.14:4838: use of closed network connection Apr 17 22:37:32.641387194 (246h51m19.460623218s) [noise=true , address=145.79.227.14:4838 , piece_id=E4PH3HQPDHCJT3H4PWPL6ZOWZYN7V45O7KA2EF44E6EUATWDE5SA, seg_pos={0 0}, upload=236994]: Long tail cancellation due to optimal threshold Apr 17 22:37:32.641406541 (246h51m19.460642565s) [seg_pos={0 0}, upload=236994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:37:32.641474403 (246h51m19.460710437s) [upload=236994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:37:32.641479482 (246h51m19.460715506s) [upload=236994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:37:32.923601653 (246h51m19.742837697s) [upload=236994]: Done uploading segments... Apr 17 22:37:32.923626632 (246h51m19.742862656s) [upload=236994]: Upload finished. err: <nil> Duration: 30.002608798s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....................XXXXX..............XXXXXXXXXX..........] duration: 13.590500343s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:37:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.590523077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [.X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XXXX.X] duration: 5.000737168s trace: https://jaeger.dev.storj.io/trace/2e32fa9041b77e48 FAIL (2026-04-17 22:36:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000757998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..........X.....X...X...X........] duration: 9.417531021s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.417556731s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XXXX.] duration: 4.872561611s trace: https://jaeger.dev.storj.io/trace/3a91c38eb639d162 FAIL (2026-04-17 22:36:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.872592801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [......................XXXXX..............XXXXXXXXXX.........] duration: 10.765775025s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:36:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.765800444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..........X.....X...X...X.......] duration: 9.409993578s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.410025138s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [...X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XXXX] duration: 5.001008061s trace: https://jaeger.dev.storj.io/trace/3a91c38eb639d162 FAIL (2026-04-17 22:35:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001053628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.......................XXXXX..............XXXXXXXXXX........] duration: 11.49320511s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:35:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.493232883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X..........X.....X...X...X......] duration: 7.849815628s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:34:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 7.849858079s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XXX] duration: 5.000825422s trace: https://jaeger.dev.storj.io/trace/30b6aa7d7c7a9547 FAIL (2026-04-17 22:34:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000862713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [........................XXXXX..............XXXXXXXXXX.......] duration: 14.267975986s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:34:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.26800373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X..........X.....X...X...X.....] duration: 8.22789674s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 8.22792236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......XX] duration: 5.001248505s trace: https://jaeger.dev.storj.io/trace/41319d32d7ca2f04 FAIL (2026-04-17 22:34:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001310003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.........................XXXXX..............XXXXXXXXXX......] duration: 21.100809557s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:34:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.100835567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..........................XXXXX..............XXXXXXXXXX.....] duration: 14.854008387s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:33:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.854031542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [......X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......X] duration: 5.001543951s trace: https://jaeger.dev.storj.io/trace/20310ded575dc3e9 FAIL (2026-04-17 22:33:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001561013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X..........X.....X...X...X....] duration: 8.09907452s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:33:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 8.099109888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X..........X.....X...X...X...] duration: 9.012881047s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:32:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.012907748s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...........................XXXXX..............XXXXXXXXXX....] duration: 23.157562201s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:32:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.157589464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.......X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X......] duration: 5.001326296s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:31:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00135452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X..........X.....X...X...X..] duration: 8.826861302s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:31:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 8.82688608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X.....] duration: 4.321096124s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:31:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.321149556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [............................XXXXX..............XXXXXXXXXX...] duration: 8.641251372s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:31:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.641267353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X..........X.....X...X...X.] duration: 9.573266348s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:30:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 9.573290803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X..........X.....X...X...X] duration: 30.004129062s trace: https://jaeger.dev.storj.io/trace/aa8dd0c2726040b FAIL (2026-04-17 22:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rting upload Apr 17 22:29:58.835057253 (246h43m45.654293278s) [upload=236876]: Uploading segments... Apr 17 22:29:58.835165883 (246h43m45.654401917s) [upload=236876]: Got next segment. Inline: false Apr 17 22:29:58.835232402 (246h43m45.654468426s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:29:58.835238574 (246h43m45.654474618s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:29:59.067488813 (246h43m45.886724837s) [seg_pos={0 0}, upload=236876]: Begin upload segment... Apr 17 22:29:59.067537137 (246h43m45.886773161s) [seg_pos={0 0}, upload=236876]: Done begin upload segment. Apr 17 22:29:59.067561012 (246h43m45.886797036s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.067830032 (246h43m45.887066056s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068075445 (246h43m45.887311469s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068380023 (246h43m45.887616047s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068651287 (246h43m45.887887321s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068700011 (246h43m45.887936036s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.068906279 (246h43m45.888142303s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069035529 (246h43m45.888271553s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069154007 (246h43m45.888390031s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069349645 (246h43m45.888585679s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069390844 (246h43m45.888626868s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069652820 (246h43m45.888888844s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069678600 (246h43m45.888914624s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.069989129 (246h43m45.889225153s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070116876 (246h43m45.88935291s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:29:59.070370365 (246h43m45.889606389s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Uploading piece... Apr 17 22:30:00.764844533 (246h43m47.584080557s) [upload=236876]: Next returned nil segment Apr 17 22:30:00.764856526 (246h43m47.58409255s) [upload=236876]: Waiting for error group managing segments... Apr 17 22:30:01.172855061 (246h43m47.992091086s) [noise=true , address=77.90.31.11:4558 , piece_id=NSDAE44NWCODFKSTGMMBQMMHQ7OYA6PZKYDYL64S72FJWGKJVN2Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.179691816 (246h43m47.99892784s) [noise=true , address=77.90.31.15:4702 , piece_id=JCFGSXZFDZPIZ4SYSKLC5D2T3RLESKYY3DJGJ3LXLDP3KYNL5UHQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.182167044 (246h43m48.001403068s) [noise=true , address=77.90.31.18:4808 , piece_id=4H5IAJP27CJKXYSN46ODSYQU5Q4353L2HQXLWEWS3NDOIU3BX3HQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:01.208595359 (246h43m48.027831383s) [noise=true , address=77.90.31.16:4738 , piece_id=FTFWZ5OBSPADFG7LFKKTQA7Y624CTKS7SP63JK4WHNIOO2HA6NEQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.184309598 (246h43m49.003545632s) [noise=true , address=145.79.227.10:4404 , piece_id=YDNHNV6GS2MBPBN33CONCVEOS4GS4FP4LMK7XZOFHDSTUNCW22VA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.316930838 (246h43m49.136166872s) [noise=true , address=145.79.227.50:4378 , piece_id=2OQWIBCEAOA5ONU4NWYQBBDG6ADMKJERRNP6HSVJITVCBCN2NPLA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.380924103 (246h43m49.200160137s) [noise=true , address=145.79.227.48:4304 , piece_id=A567AXDVMF6VSUJ5NSAXZ7ML6JT4MH2O2OXGTUXI3VJ6NTOQEJFQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.507617655 (246h43m49.326853699s) [noise=true , address=145.79.227.30:4765 , piece_id=JIAF4DZCTE6GN2UL33OGFRPAPUMKN2GFPJ67LQYEAOINFVYBT5NQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:02.753140150 (246h43m49.572376204s) [noise=true , address=145.79.227.23:4548 , piece_id=SHVRCYHOSWGGHU7IBZVH7CG2JM6SHKZMCNI44D26UHHHR34L2O6Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.055397872 (246h43m49.874633927s) [noise=true , address=45.91.116.9:4265 , piece_id=OV43AXIPOPW6OS7EVTYDDRWGGEXH2SCIHGJ4HNCSYDZ67SCUUT5Q, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.139020520 (246h43m49.958256554s) [noise=true , address=145.79.227.27:4692 , piece_id=TN5VBYYI2H4FY6D7PJIBWTKKZPKX3WHXGJ7SOYXFKIISQCEPPKHA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.364481308 (246h43m50.183717342s) [noise=true , address=45.91.116.10:4085 , piece_id=TDS7GIPQ3ZVZ373ZH4ZZO6ZZZLVW7GBYKEGCLZW7LZGFPRXHEWSQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478749280 (246h43m50.297985324s) [noise=true , address=145.79.227.16:4915 , piece_id=XAKDZAOKFT3OELXI4AS2GNDHQ3ZQ4B65UQZX6MLH5NCVMOTFTKNQ, seg_pos={0 0}, upload=236876]: Done uploading piece. err: <nil> Apr 17 22:30:03.478782654 (246h43m50.298018688s) [seg_pos={0 0}, upload=236876]: Satisfied cohort requirements after 13 pieces Apr 17 22:30:03.479062133 (246h43m50.298298167s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JBXGbBeXrAvFpBao6RroNwiAP4wk4wWWkhAaCcUUatGa2k9Ev): protocol: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57260->45.91.116.5:4332: use of closed network connection Apr 17 22:30:03.479072803 (246h43m50.298308837s) [noise=true , address=45.91.116.5:4332 , piece_id=YJ7FJIPRXYZG3IAKZDMQTNOATF4P7V2WT7NG5ZIVWWFKJ6QDO3QA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479703762 (246h43m50.298939796s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:55638->145.79.227.2:4115: use of closed network connection; EOF Apr 17 22:30:03.479717217 (246h43m50.298953251s) [noise=true , address=145.79.227.2:4115 , piece_id=435MYJTT3DWFL77EHSDEQ3KHEML5KDYFI3DMUKD3PN7DGAGOM3XA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479939647 (246h43m50.299175681s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:30:03.479950778 (246h43m50.299186812s) [noise=true , address=145.79.227.35:4486 , piece_id=LNL4OISGTACL2VYR7IFJLFGWCI56LPUUWJLQ7L46YLQ6GJSVJVTA, seg_pos={0 0}, upload=236876]: Long tail cancellation due to optimal threshold Apr 17 22:30:03.479978602 (246h43m50.299214636s) [seg_pos={0 0}, upload=236876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:30:03.480082452 (246h43m50.299318486s) [upload=236876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:30:03.480092381 (246h43m50.299328415s) [upload=236876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:30:03.752804201 (246h43m50.572040246s) [upload=236876]: Done uploading segments... Apr 17 22:30:03.752830271 (246h43m50.572066305s) [upload=236876]: Upload finished. err: <nil> Duration: 30.00415391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X....] duration: 4.103372891s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.103400545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................XXXXX..............XXXXXXXXXX..] duration: 10.464943354s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:30:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.464968824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X..........X.....X...X...] duration: 7.826480049s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 7.826502211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X...] duration: 4.434090759s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.434166285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..............................XXXXX..............XXXXXXXXXX.] duration: 11.710016229s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:29:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.710081165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X..........X.....X...X..] duration: 9.151741779s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:28:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 9.151758222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [...........X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X..] duration: 4.620470495s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.620520199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...............................XXXXX..............XXXXXXXXXX] duration: 1m0.000997603s trace: https://jaeger.dev.storj.io/trace/51ac1c35b102be61 FAIL (2026-04-17 22:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001033192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X.] duration: 4.403269531s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.40329511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X..........X.....X...X.] duration: 7.65982004s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:28:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 7.659893341s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.X] duration: 5.000990072s trace: https://jaeger.dev.storj.io/trace/45da994cfc6a0caa FAIL (2026-04-17 22:28:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001007857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [................................XXXXX..............XXXXXXXXX] duration: 1m0.001720016s trace: https://jaeger.dev.storj.io/trace/532f681695e7f03 FAIL (2026-04-17 22:27:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001738201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X..........X.....X...X] duration: 30.00147431s trace: https://jaeger.dev.storj.io/trace/e030e7ccfe74169 FAIL (2026-04-17 22:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 37959 (246h40m43.264673983s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:26:56.445442749 (246h40m43.264678783s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:26:56.680984391 (246h40m43.500220415s) [seg_pos={0 0}, upload=236828]: Begin upload segment... Apr 17 22:26:56.681048645 (246h40m43.50028469s) [seg_pos={0 0}, upload=236828]: Done begin upload segment. Apr 17 22:26:56.681083343 (246h40m43.500319387s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681097750 (246h40m43.500333784s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681203023 (246h40m43.500439047s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.681979182 (246h40m43.501215226s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.682540013 (246h40m43.501776047s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.683355678 (246h40m43.502591702s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684087060 (246h40m43.503323084s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.684591082 (246h40m43.503827106s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.686358003 (246h40m43.505594027s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.688665065 (246h40m43.507901089s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.690702617 (246h40m43.509938652s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.692496279 (246h40m43.511732303s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.695655366 (246h40m43.51489139s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.697242800 (246h40m43.516478834s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.699718578 (246h40m43.518954612s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:56.701391005 (246h40m43.52062703s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Uploading piece... Apr 17 22:26:58.503644796 (246h40m45.32288082s) [upload=236828]: Next returned nil segment Apr 17 22:26:58.503661007 (246h40m45.322897041s) [upload=236828]: Waiting for error group managing segments... Apr 17 22:26:58.806983955 (246h40m45.626219979s) [noise=true , address=77.90.31.15:4702 , piece_id=O773XFC766WC2Z7UAOQ7LQ7SP5DHDDFHPHKNQ7MNZROPZU2EEYLA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:58.825413052 (246h40m45.644649076s) [noise=true , address=77.90.31.19:4842 , piece_id=YZJD6TU2IJFMMLKQA4QGL3VXVDZ2PMLXYYYCYMOG657J7FCNYGRQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.698754768 (246h40m46.517990792s) [noise=true , address=145.79.227.24:4586 , piece_id=RVEATNQFW32IPUX46EXJO3Z736HCD4JDUWXQSKB74YIW44LO6EOQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.942061167 (246h40m46.761297211s) [noise=true , address=145.79.227.46:4606 , piece_id=SUBVZCBCNTMYYSMPSHB4GN62F3TVN6E6E7BZ4JRR7MJWKBWRLNKA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.989468271 (246h40m46.808704305s) [noise=true , address=45.91.116.4:4953 , piece_id=T4HXGXM44IIX72NJT56ZOHY7XL6TOONTKE6HDRJAVJLGCSMJL3JA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:26:59.991320666 (246h40m46.81055669s) [noise=true , address=145.79.227.50:4379 , piece_id=PTVAYXM7PBQQQYWTUFKBVFNJ57HGELBC35IQNZYBNM5L2NJPGMBQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.371241425 (246h40m47.190477449s) [noise=true , address=145.79.227.15:4875 , piece_id=SV5NW4ICDLDOCFLE5T4YDTANVULF5R2KN5GNTNVOEA65MEJDYG4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.410211514 (246h40m47.229447538s) [noise=true , address=45.91.116.9:4264 , piece_id=7YYEPUQA2NRFESVJW7EMCK57JLXQ3HJI7YXE5TAI53P3NRQ5Q5DQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.443769760 (246h40m47.263005784s) [noise=true , address=45.91.116.14:4190 , piece_id=47PFTEBQCHBZROTCASR4IH6VE3OY22XHHSEOLR32UB47QHP5R6WA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.574700229 (246h40m47.393936253s) [noise=true , address=145.79.227.51:4415 , piece_id=WKS3G22HDLTVBOKJP5CJ3ITXAIJLZRT5LOZPUJXH3QA5VPK64P3Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.616364549 (246h40m47.435600603s) [noise=true , address=145.79.227.26:4654 , piece_id=VUZBEZF4AA75XOYIWCYERPF2FZNCMY77I36VVBRVTYEBA6NF2V4Q, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:00.966164496 (246h40m47.78540051s) [noise=true , address=145.79.227.6:4263 , piece_id=LHL6EOO342M52QIFMQTUEUEY52I4RR52SLX2OOUZ3PIRR6DXIYIQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447637825 (246h40m48.266873859s) [noise=true , address=45.91.116.13:4156 , piece_id=73ML6QIR5G2TDEN43XJSSNBB7LCE6QAV3TKLCB2GNWSYP6LWNEXQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: <nil> Apr 17 22:27:01.447658094 (246h40m48.266894118s) [seg_pos={0 0}, upload=236828]: Satisfied cohort requirements after 13 pieces Apr 17 22:27:01.447856296 (246h40m48.26709232s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51190->145.79.227.2:4115: use of closed network connection Apr 17 22:27:01.447865153 (246h40m48.267101187s) [noise=true , address=145.79.227.2:4115 , piece_id=OL5SBU3C4HBWWMPCXCJCCNSAMXQRWO7QW4BDBLSFTKUTW54N7BUQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.447882577 (246h40m48.267118601s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51438->145.79.227.9:4366: use of closed network connection Apr 17 22:27:01.447890883 (246h40m48.267126907s) [noise=true , address=145.79.227.9:4366 , piece_id=FIJURSIIEGW3Q7TZQWBM74L2VG5O3Z573MOMZZS57MCNC2QUPAFA, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448081771 (246h40m48.267317805s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz): protocol: write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection; write tcp 188.166.225.170:52426->145.79.227.12:4770: use of closed network connection Apr 17 22:27:01.448088484 (246h40m48.267324518s) [noise=true , address=145.79.227.12:4770 , piece_id=4KEGEGGSRM6XM2HFBL4DXHRVAO4WB7NIRP5YIVPCZPH7QZS4VCDQ, seg_pos={0 0}, upload=236828]: Long tail cancellation due to optimal threshold Apr 17 22:27:01.448108674 (246h40m48.267344698s) [seg_pos={0 0}, upload=236828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:27:01.448191924 (246h40m48.267427958s) [upload=236828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:27:01.448197464 (246h40m48.267433488s) [upload=236828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:27:01.723713096 (246h40m48.542949141s) [upload=236828]: Done uploading segments... Apr 17 22:27:01.723737243 (246h40m48.542973277s) [upload=236828]: Upload finished. err: <nil> Duration: 30.001506042s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X.] duration: 4.318718346s trace: https://jaeger.dev.storj.io/trace/366a6dd433456773 FAIL (2026-04-17 22:27:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318745969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...X] duration: 5.001556725s trace: https://jaeger.dev.storj.io/trace/366a6dd433456773 FAIL (2026-04-17 22:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X..........X.....X...] duration: 18.900543393s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:26:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 18.900715194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.................................XXXXX..............XXXXXXXX] duration: 1m0.000797277s trace: https://jaeger.dev.storj.io/trace/19fe5fe1f7c99e30 FAIL (2026-04-17 22:26:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000809669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X..........X.....X..] duration: 7.81162255s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 7.811662106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X...] duration: 4.621319847s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:25:42 UTC) orbiter warning: context deadline exceeded Duration: 4.621344313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [..................................XXXXX..............XXXXXXX] duration: 1m0.00082905s trace: https://jaeger.dev.storj.io/trace/30f9dc36da647e3c FAIL (2026-04-17 22:25:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00084001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X..........X.....X.] duration: 9.351278529s trace: https://jaeger.dev.storj.io/trace/42b424317d5a4121 FAIL (2026-04-17 22:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:24:05.694992332 (246h37m52.514228356s) [upload=236781]: Starting upload Apr 17 22:24:05.695045855 (246h37m52.514281879s) [upload=236781]: Done starting upload Apr 17 22:24:05.695147542 (246h37m52.514383566s) [upload=236781]: Uploading segments... Apr 17 22:24:05.695218739 (246h37m52.514454773s) [upload=236781]: Got next segment. Inline: false Apr 17 22:24:05.695247164 (246h37m52.514483188s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:24:05.695251513 (246h37m52.514487537s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:24:05.932814858 (246h37m52.752050872s) [seg_pos={0 0}, upload=236781]: Begin upload segment... Apr 17 22:24:05.932862179 (246h37m52.752098203s) [seg_pos={0 0}, upload=236781]: Done begin upload segment. Apr 17 22:24:05.932878050 (246h37m52.752114074s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.932886456 (246h37m52.75212248s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933244698 (246h37m52.752480722s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933248785 (246h37m52.752484809s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933532453 (246h37m52.752768477s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933584163 (246h37m52.752820187s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.933898440 (246h37m52.753134464s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934250218 (246h37m52.753486232s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934363717 (246h37m52.753599741s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934643347 (246h37m52.753879371s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.934869704 (246h37m52.754105728s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935128173 (246h37m52.754364197s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935374137 (246h37m52.754610161s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935623228 (246h37m52.754859252s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.935978473 (246h37m52.755214497s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:05.936271699 (246h37m52.755507723s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Uploading piece... Apr 17 22:24:07.749332052 (246h37m54.568568076s) [upload=236781]: Next returned nil segment Apr 17 22:24:07.749345899 (246h37m54.568581923s) [upload=236781]: Waiting for error group managing segments... Apr 17 22:24:08.055489009 (246h37m54.874725033s) [noise=true , address=77.90.31.21:4880 , piece_id=6R2XX355MHAQOOADHD5CEGMUVC5FWVYGVKFNX7C2O325LZT2DYBA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.060851439 (246h37m54.880087463s) [noise=true , address=77.90.31.11:4558 , piece_id=AWPPBKGMKOHLRA5NE5TOWGK2SWRYG6V26A5IKO75STCG3WLG2IEQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:08.087574814 (246h37m54.906810838s) [noise=true , address=77.90.31.19:4841 , piece_id=NDGKANS3OTNP2QBIVLRQ4BHGEBEFKFMWA55YTPLVC2VNJGNW3JYQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.218312049 (246h37m56.037548073s) [noise=true , address=45.91.116.5:4332 , piece_id=QZTQO33Y255MFAURSMSJ2QKMLUBMQTU6H6RTDQNIGITS2WRRZSEA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.516129322 (246h37m56.335365346s) [noise=true , address=145.79.227.18:4009 , piece_id=3P5LZAABYMRZ472BTGVLEQAIQT7ZGJHQGUJH3OQGSPVQ7ZM6MTVQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.574962518 (246h37m56.394198532s) [noise=true , address=145.79.227.22:4514 , piece_id=CRUQJJWKPX3UOGJSW36ET66Q3Y5NYTBI76S7ZJM6PJU4TMHEMDJA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.644416080 (246h37m56.463652104s) [noise=true , address=145.79.227.35:4483 , piece_id=3YYCQB3ZSKJDOSVIE2CLNRZ34FKOQOMEL6C2WXS5Z7U4ZAL4AMFQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.696489518 (246h37m56.515725532s) [noise=true , address=145.79.227.2:4116 , piece_id=NPA4Q6IW6WYNEFGYL5QSYL6ATXOSETYZF6ASPD6NAYQ4FW5KPX3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:09.985990091 (246h37m56.805226105s) [noise=true , address=145.79.227.50:4377 , piece_id=F2X3PG6K3WEM7QIK3OSVAQCGZ4O445SV4ZE73NJWCHCWNUTKT73A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.022897720 (246h37m56.842133744s) [noise=true , address=145.79.227.38:4977 , piece_id=7YU7TSHB4UJKZERR3LZXZR4HRXRXWPSXZPUVCF5Q2XOX5R7O4OZQ, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.034800812 (246h37m56.854036836s) [noise=true , address=45.91.116.14:4190 , piece_id=4O3O2IOEL4TBVLUVW4LVNGTW3QG6GD4IJDUCIWKIYS4OLRF5Q4XA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.120044970 (246h37m56.939280994s) [noise=true , address=45.91.116.7:4012 , piece_id=PQ7QW3WAIJ3S73MLBKMOPU5DGTRSQJLKYNLMLTOJDKYGIDKJZD3A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133421156 (246h37m56.95265721s) [noise=true , address=45.91.116.9:4264 , piece_id=DPOESBBAQZ6A6MRIGWABZ5YQEJAASGCUA7QOXX7YLFNL7LSAG57A, seg_pos={0 0}, upload=236781]: Done uploading piece. err: <nil> Apr 17 22:24:10.133446624 (246h37m56.952682729s) [seg_pos={0 0}, upload=236781]: Satisfied cohort requirements after 13 pieces Apr 17 22:24:10.133736224 (246h37m56.952972328s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133750301 (246h37m56.952986405s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kn3uD4MCE56bN4MYYsGjDJd2qebgZ6VB6mavL711M8hoU8fbE): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133763746 (246h37m56.95299977s) [noise=true , address=145.79.227.23:4550 , piece_id=YSQSAAUEXWUV5DK56N2MCZUILBGTM7DKHZSWI5DZZ7DAESRNV6YA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133825065 (246h37m56.953061149s) [noise=true , address=145.79.227.34:4906 , piece_id=GJV7FAUX7I5WPEC6X5MCLKUIUBC36ASOE5U74YX6R7GL5Q5K3Z3Q, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133966618 (246h37m56.953202662s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Apr 17 22:24:10.133976136 (246h37m56.953212161s) [noise=true , address=145.79.227.19:4078 , piece_id=2FJ5HPGT25ODB36BIOXVD5FP6Q2B34SISA3FO3SVOWZRKDTC74VA, seg_pos={0 0}, upload=236781]: Long tail cancellation due to optimal threshold Apr 17 22:24:10.133995313 (246h37m56.953231348s) [seg_pos={0 0}, upload=236781]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:24:10.134105837 (246h37m56.953341861s) [upload=236781]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:24:10.134112840 (246h37m56.953348864s) [upload=236781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:24:10.433095060 (246h37m57.252331084s) [upload=236781]: Done uploading segments... Apr 17 22:24:10.433115701 (246h37m57.252351735s) [upload=236781]: Upload finished. err: <nil> Duration: 9.351302906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [....X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.X..] duration: 4.482347838s trace: https://jaeger.dev.storj.io/trace/579efa8346903a39 FAIL (2026-04-17 22:24:43 UTC) orbiter warning: context deadline exceeded Duration: 4.482381623s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)