Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.00071503s trace: https://jaeger.dev.storj.io/trace/4d490dc355399381 FAIL (2025-11-15 16:05:27 UTC) orbiter warning: context deadline exceeded Duration: 5.000738895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X.......X...........................] duration: 6.364739909s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:05:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.364772852s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 12.325105196s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:05:10 UTC) orbiter warning: context deadline exceeded Duration: 12.325136015s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....X.......X..........................] duration: 6.184081378s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:04:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.184111586s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 12.532851288s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:04:35 UTC) orbiter warning: context deadline exceeded Duration: 12.532883669s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 12.29386856s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:03:58 UTC) orbiter warning: context deadline exceeded Duration: 12.293890963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....X.......X.........................] duration: 6.004200764s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:03:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.004229699s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 11.818057945s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:03:36 UTC) orbiter warning: context deadline exceeded Duration: 11.818163025s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 15.038183553s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:02:51 UTC) orbiter warning: context deadline exceeded Duration: 15.038210574s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....................X.....X.......X........................] duration: 6.200709636s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:02:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.20073775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....................X.....X.......X.......................] duration: 6.179201129s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:02:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.179221177s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.937574894s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:01:49 UTC) orbiter warning: context deadline exceeded Duration: 6.937605703s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.875671075s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:01:01 UTC) orbiter warning: context deadline exceeded Duration: 8.875703147s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....................X.....X.......X......................] duration: 6.171317611s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:00:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.171347459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 17.142236485s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:00:49 UTC) orbiter warning: context deadline exceeded Duration: 17.142267955s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X....................X.....X.......X.....................] duration: 6.31653791s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.317473381s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 10.224419984s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:59:53 UTC) orbiter warning: context deadline exceeded Duration: 10.224450914s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 11.793686569s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:59:38 UTC) orbiter warning: context deadline exceeded Duration: 11.793703741s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....................X.....X.......X....................] duration: 5.929551885s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:59:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 5.929878918s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.168348155s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:58:44 UTC) orbiter warning: context deadline exceeded Duration: 7.16837138s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X....................X.....X.......X...................] duration: 6.158688715s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:58:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.158720425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X....................X.....X.......X..................] duration: 6.097304887s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:57:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.097331366s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 12.359863343s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:57:40 UTC) orbiter warning: context deadline exceeded Duration: 12.359934438s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....................X.....X.......X.................] duration: 6.923421246s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.923454169s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 12.879656456s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:57:04 UTC) orbiter warning: context deadline exceeded Duration: 12.879787935s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....................X.....X.......X................] duration: 6.361151304s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:56:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.361179907s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 7.094758635s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:56:30 UTC) orbiter warning: context deadline exceeded Duration: 7.094785957s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X.....X.......X...............] duration: 6.003000589s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:56:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.003025506s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 11.782807624s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:55:51 UTC) orbiter warning: context deadline exceeded Duration: 11.782834175s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.378989648s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:55:19 UTC) orbiter warning: context deadline exceeded Duration: 7.379018243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X....................X.....X.......X..............] duration: 6.324767698s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:55:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.324941378s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 13.306573315s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:54:47 UTC) orbiter warning: context deadline exceeded Duration: 13.30660175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X....................X.....X.......X.............] duration: 7.354795225s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:54:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 7.354819161s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.245343336s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:54:13 UTC) orbiter warning: context deadline exceeded Duration: 6.245386969s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X....................X.....X.......X............] duration: 5.954836653s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:54:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 5.954867692s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.654289738s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:53:54 UTC) orbiter warning: context deadline exceeded Duration: 11.654320968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X....................X.....X.......X...........] duration: 6.097277534s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.097303294s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.376252263s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:52:49 UTC) orbiter warning: context deadline exceeded Duration: 7.376284073s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X....................X.....X.......X..........] duration: 6.199012625s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.1990414s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.523892164s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:52:19 UTC) orbiter warning: context deadline exceeded Duration: 8.524014977s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X....................X.....X.......X.........] duration: 6.196695189s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:51:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.196724143s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 12.224797995s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:51:31 UTC) orbiter warning: context deadline exceeded Duration: 12.224878288s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X....................X.....X.......X........] duration: 6.558253442s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.558279033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X....................X.....X.......X.......] duration: 6.323735924s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:50:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.323764237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 16.209515758s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:50:25 UTC) orbiter warning: context deadline exceeded Duration: 16.209543981s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 19.230667912s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:49:59 UTC) orbiter warning: context deadline exceeded Duration: 19.230689663s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X....................X.....X.......X......] duration: 6.03506555s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.035096007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 14.452687753s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 15:48:46 UTC) orbiter warning: context deadline exceeded Duration: 14.452715605s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....................X.....X.......X.....] duration: 6.314101907s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:48:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.314138898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X....................X.....X.......X....] duration: 6.215267526s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 15:48:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.215309226s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)