Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X....................X.....X.......X.......] duration: 6.323735924s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:50:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.323764237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 16.209515758s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:50:25 UTC) orbiter warning: context deadline exceeded Duration: 16.209543981s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 19.230667912s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:49:59 UTC) orbiter warning: context deadline exceeded Duration: 19.230689663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X....................X.....X.......X......] duration: 6.03506555s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.035096007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 14.452687753s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:48:46 UTC) orbiter warning: context deadline exceeded Duration: 14.452715605s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....................X.....X.......X.....] duration: 6.314101907s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:48:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.314138898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X....................X.....X.......X....] duration: 6.215267526s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:48:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.215309226s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 7.771622765s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:47:47 UTC) orbiter warning: context deadline exceeded Duration: 7.771635689s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 21.001438901s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:47:39 UTC) orbiter warning: context deadline exceeded Duration: 21.001467345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X....................X.....X.......X...] duration: 6.906674147s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.906703634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X....................X.....X.......X..] duration: 6.166168649s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.166204097s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.971897285s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:46:24 UTC) orbiter warning: context deadline exceeded Duration: 6.971971105s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....................X.....X.......X.] duration: 6.012749s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:46:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.012775491s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.319909049s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:46:10 UTC) orbiter warning: context deadline exceeded Duration: 21.319936902s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X....................X.....X.......X] duration: 30.004651195s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 30.00472168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 17.024214904s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:45:05 UTC) orbiter warning: context deadline exceeded Duration: 17.024326677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....................X.....X.......] duration: 7.619223675s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 7.61925717s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.842658928s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:43:51 UTC) orbiter warning: context deadline exceeded Duration: 6.842682794s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....................X.....X......] duration: 5.979692297s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.979720721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....................X.....X.....] duration: 6.227682957s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:43:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.227776025s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.743181503s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:42:54 UTC) orbiter warning: context deadline exceeded Duration: 8.743210558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....................X.....X....] duration: 6.091889339s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:42:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.091920236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....................X.....X...] duration: 6.901869251s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.901898266s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 17.340234157s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:41:59 UTC) orbiter warning: context deadline exceeded Duration: 17.34669001s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....................X.....X..] duration: 6.255959129s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:41:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.255988856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....................X.....X.] duration: 6.31852559s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:41:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.318547483s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 11.427360713s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:40:57 UTC) orbiter warning: context deadline exceeded Duration: 11.427376613s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....................X.....X] duration: 30.349691511s trace: https://jaeger.dev.storj.io/trace/510d40671a2e16db FAIL (2025-11-15 15:40:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 99685s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.379650845 (151h35m59.931121306s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380175003 (151h35m59.931645504s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.380597638 (151h35m59.932068149s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381093533 (151h35m59.932563994s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.381549842 (151h35m59.933020303s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382050605 (151h35m59.933521066s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.382577869 (151h35m59.93404833s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.383396578 (151h35m59.934867059s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.384641509 (151h35m59.93611198s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385075385 (151h35m59.936545846s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.385582150 (151h35m59.937052611s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.390418980 (151h35m59.941889471s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:01.393183775 (151h35m59.944654236s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Uploading piece... Nov 15 15:40:03.291550401 (151h36m1.843020872s) [upload=133501]: Next returned nil segment Nov 15 15:40:03.291571852 (151h36m1.843042334s) [upload=133501]: Waiting for error group managing segments... Nov 15 15:40:03.741925896 (151h36m2.293396347s) [noise=true , address=77.90.31.16:4734 , piece_id=U7PPFNPZSQLVKELCI7ZHEFNMNIWDNMKYO5M3JPHF4ZGSSAOQ3TDA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759113135 (151h36m2.310583596s) [noise=true , address=77.90.31.19:4843 , piece_id=GOEBCSUAO5I6X2K6AUOGH2X4FDINXL6YMHZ75KDUAOQLRV3FIOAQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.759879684 (151h36m2.311350145s) [noise=true , address=45.91.116.9:4262 , piece_id=3G4IWETV2SM4RU6HTKZTZBZCI2DQEZIKL7K33756DTXFNUJWVXOA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:03.761046266 (151h36m2.312516737s) [noise=true , address=45.91.116.10:4081 , piece_id=7YSL3PTXRPXFT4U47Z5PKHV7OGRGE5RCULNYXNAP6JRHIO7XF3DQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:04.695841005 (151h36m3.247311466s) [noise=true , address=77.90.31.21:4878 , piece_id=Q4L5FOUEZMKTRNK2CNG5VLI5RRPLRY7UGS5DFZACWTPUYBHISDYA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:05.081073404 (151h36m3.632543865s) [noise=true , address=45.91.116.11:4117 , piece_id=LBQ2HMPGZDU53G7SFWZG43PUTDQCM5FNCJRFLVKBFUDVQRIJUJVA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: <nil> Nov 15 15:40:30.365495123 (151h36m28.916965574s) [noise=true , address=145.79.227.10:4401 , piece_id=PHSWUHUSQ3NNHHCXQKAQJJXTTWTGICJL4SY2U5HOB7H6QZGMTWHQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: rpc: dial tcp 145.79.227.10:4401: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.368875210 (151h36m28.920345672s) [noise=true , address=145.79.227.8:4331 , piece_id=YD5UGAKAKKZ4CFWWF4ERKVFGOGRLVACF7DDN4DPXKTMA3TWF22KA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47804->145.79.227.8:4331: use of closed network connection Nov 15 15:40:30.369094428 (151h36m28.920564889s) [noise=true , address=145.79.227.13:4803 , piece_id=BAYZRPZKPWAV4M6CZPF77IDBXRCU6OJV5BPGCRKNI2KZD7VXPHLA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49408->145.79.227.13:4803: use of closed network connection Nov 15 15:40:30.369436439 (151h36m28.92090691s) [noise=true , address=145.79.227.27:4690 , piece_id=C7DEHMBYOW7ZLIYLNXVQQRWFVWJGKKVUM4CLV5CIR4EOG7O74XEA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46972->145.79.227.27:4690: use of closed network connection Nov 15 15:40:30.369688138 (151h36m28.921158589s) [noise=true , address=145.79.227.22:4511 , piece_id=3SQEBWT6E6DGXFXHZZNPLDGPBF2KTE45HIZZTGSSU2AGLN3G5IJQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:53204->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.369959295 (151h36m28.921429766s) [noise=true , address=145.79.227.23:4545 , piece_id=Y6PAPNHTU3KELQSHBHXE3I2ZGOURY3E3ULI25EC56IBCIOGKOSSQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:56414->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370144076 (151h36m28.921614537s) [noise=true , address=145.79.227.25:4617 , piece_id=DVGB3HMHVA2DPMOGCKDN2UKGHBM4PI56ZH5IXXCXSYHQQJPMKOKA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: read tcp 188.166.225.170:46604->145.79.227.25:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370371810 (151h36m28.921842271s) [noise=true , address=145.79.227.17:4971 , piece_id=J5ERHQCXSGAWONB5MMFEUB5V3ZIJN7MXA3MGPDHGKX6IMS7IGUWQ, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: read tcp 188.166.225.170:40538->145.79.227.17:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370508951 (151h36m28.921979403s) [noise=true , address=145.79.227.32:4834 , piece_id=3DRTLDS5TVHEIOMA7B7ZSPKDMJR5SYLSNPCGWJU4L44RELGS577Q, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:55744->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370686640 (151h36m28.922157101s) [noise=true , address=145.79.227.18:4005 , piece_id=CF5BYHNMHYL6SK6AUC6WKMIVOO657V7ONH3O4RLCPDOJ6MBASMBA, seg_pos={0 0}, upload=133501]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:43884->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:40:30.370713732 (151h36m28.922184193s) [seg_pos={0 0}, upload=133501]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:40:30.680582871 (151h36m29.232053332s) [upload=133501]: Done uploading segments... Nov 15 15:40:30.680658977 (151h36m29.232129428s) [upload=133501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.349718372s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 18.06860374s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:39:51 UTC) orbiter warning: context deadline exceeded Duration: 18.068630261s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....................X.....] duration: 6.324648926s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:39:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.324676739s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 16.050928283s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:38:35 UTC) orbiter warning: context deadline exceeded Duration: 16.050950705s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....................X....] duration: 6.069104638s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:38:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.069133614s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 14.302733828s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:37:59 UTC) orbiter warning: context deadline exceeded Duration: 14.30277205s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 14.487719664s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:37:35 UTC) orbiter warning: context deadline exceeded Duration: 14.487745953s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....................X...] duration: 5.972485599s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.972509065s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 14.702622345s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:36:46 UTC) orbiter warning: context deadline exceeded Duration: 14.702649887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....................X..] duration: 6.025635802s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.025665527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....................X.] duration: 6.191644283s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:35:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.191707192s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 15.020690611s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:35:50 UTC) orbiter warning: context deadline exceeded Duration: 15.0209957s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....................X] duration: 30.303457554s trace: https://jaeger.dev.storj.io/trace/2fb936abd3d9d869 FAIL (2025-11-15 15:35:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 7VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902141861 (151h30m36.453612322s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.902586168 (151h30m36.454056619s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903025284 (151h30m36.454495745s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903386171 (151h30m36.454856632s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.903742249 (151h30m36.45521271s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904061607 (151h30m36.455532068s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904379293 (151h30m36.455849754s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.904677510 (151h30m36.456147961s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905073775 (151h30m36.456544236s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905420776 (151h30m36.456891227s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.905710106 (151h30m36.457180568s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906038712 (151h30m36.457509173s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906380483 (151h30m36.457850944s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.906739186 (151h30m36.458209638s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:37.907023618 (151h30m36.458494079s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Uploading piece... Nov 15 15:34:39.755563644 (151h30m38.307034106s) [upload=133421]: Next returned nil segment Nov 15 15:34:39.755598581 (151h30m38.307069042s) [upload=133421]: Waiting for error group managing segments... Nov 15 15:34:40.049371090 (151h30m38.600841551s) [noise=true , address=45.91.116.7:4010 , piece_id=3NCF6WRWWY64VCFHPHTMI5ZV7ERM4KRZO22YRKTNUTN46PPG7WAQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.078204877 (151h30m38.629675338s) [noise=true , address=45.91.116.4:4940 , piece_id=QUMWES7Q3APU7IS6KAC64UIJZEBS6HCGKMOUEGNDBVSQHQHEK4DQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.365578340 (151h30m38.917048801s) [noise=true , address=77.90.31.15:4697 , piece_id=XNIOCYOKVQUL3FKIDVYKGXUEM6Y57VBRQROW5UVFWKDIVUOFVUTQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.585172607 (151h30m39.136643068s) [noise=true , address=45.91.116.6:4975 , piece_id=YRUAYBMYZMEHDS7TCLMFWGC4FA5MFERA3O6ZMRSZKOBK4GU2SA3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.637582276 (151h30m39.189052747s) [noise=true , address=77.90.31.14:4661 , piece_id=N3RAJR4KTPK2WOU5CP66QE43UF5XDJYBU2XSIAW6JWYMCDXJXE6Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:34:40.969278622 (151h30m39.520749083s) [noise=true , address=77.90.31.17:4772 , piece_id=344QILI6RCFAEGS7OMJ5LCFATZICN3HJBMFRGA3HVJLLJNIOEQFA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil> Nov 15 15:35:06.687030959 (151h31m5.23850147s) [noise=true , address=145.79.227.28:4725 , piece_id=G43WYYBGJ4YZLAOGOE4P3Z6RUZUA2NW5FMA2DUHKG2S7HDT7WX5A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: rpc: dial tcp 145.79.227.28:4725: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687734048 (151h31m5.23920451s) [noise=true , address=145.79.227.16:4909 , piece_id=PY5KXWHF7T6RC54KHSQKEDM3XCF2NLUA5KXP5GZB4SJRUNEMTSIQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: rpc: dial tcp 145.79.227.16:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.687880397 (151h31m5.239350868s) [noise=true , address=145.79.227.32:4836 , piece_id=42OY7NBMWHCZNXSYH5QIKJN43V27DAEMBX23ED5KT45CJ2QBUIEA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59176->145.79.227.32:4836: use of closed network connection Nov 15 15:35:06.688182071 (151h31m5.239652572s) [noise=true , address=145.79.227.31:4799 , piece_id=4J5UFT5B4HPITXUSELHSPTASJQ7ATMR4OERGNOQVMBQZ2VSTZOHQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:55698->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.688388143 (151h31m5.239858614s) [noise=true , address=145.79.227.24:4581 , piece_id=ROJ53N5YKE6IR2ZZ5XJVPJ4VRBYTTXBBUWHTH6YQ2MB4BBBL5UUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:33378->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689347931 (151h31m5.240818392s) [noise=true , address=145.79.227.22:4511 , piece_id=RXHAMULPYAPQVT65CECXZPHCPMWDXOXU3EIYMQKSCQZPNCLPSTQA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:38432->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.689730430 (151h31m5.241200901s) [noise=true , address=145.79.227.26:4655 , piece_id=K56E4MOI3ZGAJDOYIZVJW7SHDBQD22NKPM2WM2LZ3QSR25YYOR3Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: read tcp 188.166.225.170:60878->145.79.227.26:4655: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.691048480 (151h31m5.242518941s) [noise=true , address=145.79.227.14:4837 , piece_id=THMPGVRVVIVGHE4FWGEHIVIAD46PMSSNW7CCJTJYUVGJHUFLPDBQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:60992->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692525431 (151h31m5.243995913s) [noise=true , address=145.79.227.30:4763 , piece_id=EMAYUWP22JJHMYYXDCEWZSWGNGOMY2ZIJWY7CDDHFYKUDDER5UEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36702->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692891319 (151h31m5.24436178s) [noise=true , address=145.79.227.15:4874 , piece_id=PR7W4G4AIHF6OI2B2DF6TX733V6F7MUU56FRDLDITR3YMWVDDTJA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:45554->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:35:06.692953708 (151h31m5.244424169s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Nov 15 15:35:06.987827483 (151h31m5.539297935s) [upload=133421]: Done uploading segments... Nov 15 15:35:06.987877279 (151h31m5.53934774s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.303483905s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 14.602698247s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:35:00 UTC) orbiter warning: context deadline exceeded Duration: 14.602775723s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 17.944803795s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:34:23 UTC) orbiter warning: context deadline exceeded Duration: 17.944848259s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.033088477s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:33:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.033120627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.239040935s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:33:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.239081152s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.672770875s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:33:18 UTC) orbiter warning: context deadline exceeded Duration: 15.672865556s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.287768525s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:33:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.287812881s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 19.001587675s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:33:00 UTC) orbiter warning: context deadline exceeded Duration: 19.001618745s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.........................................X.................] duration: 6.045472529s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:32:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.045500872s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.438169617s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:32:17 UTC) orbiter warning: context deadline exceeded Duration: 14.438185466s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001275756s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:32:03 UTC) orbiter warning: context deadline exceeded Duration: 30.00129921s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)