Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X........X.......................] duration: 3.604913704s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:30:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604964121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X........X......................] duration: 3.452997313s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:29:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45302213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X........X.....................] duration: 3.452328887s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:29:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452359466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X........X....................] duration: 3.439848734s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:28:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439876549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X........X...................] duration: 3.546056268s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:27:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546083751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X........X..................] duration: 3.706731771s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:26:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.706749605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.021746296s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:26:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.02177483s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X........X.................] duration: 3.501214997s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:25:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501246036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.077609058s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.077633044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X........X................] duration: 5.001023357s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:25:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001045359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X........X...............] duration: 3.62950667s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:24:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629541815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.050993124s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 7.0510329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X........X..............] duration: 3.445438828s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:24:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445466892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.219256683s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.219283354s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X........X.............] duration: 3.513857015s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:23:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513882482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X........X............] duration: 3.514397784s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:22:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514429565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.212708945s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.212750444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.260588249s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:22:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.260621151s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X........X...........] duration: 3.580139223s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:21:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580165041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.964548618s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:21:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 5.964580658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.177645766s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:21:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.177679169s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X........X..........] duration: 3.467056573s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:20:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467121476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......X...................................................] duration: 6.107046007s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:20:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.107086885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......X..................................................] duration: 6.048743655s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.048774765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X........X.........] duration: 4.431244056s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:19:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.431286287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......X.................................................] duration: 6.021754711s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:19:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.021785359s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X........X........] duration: 3.476424661s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:19:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476463497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......X................................................] duration: 6.189698187s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:18:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.189741859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.......X...............................................] duration: 7.301545126s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 7.301571147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X........X.......] duration: 3.439880955s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:18:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439909648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X........X......] duration: 3.473836357s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:17:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.473857097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.......X..............................................] duration: 6.254763783s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.254792249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....X.......X.............................................] duration: 6.158732438s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:17:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.158763607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X........X.....] duration: 5.00138404s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:16:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001410351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....X.......X............................................] duration: 6.103891729s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.103921306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.....X.......X...........................................] duration: 6.122372253s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:16:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.122404976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X........X....] duration: 3.575250612s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:15:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57528095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....X.......X..........................................] duration: 6.104516628s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.104548468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X........X...] duration: 3.615710176s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:14:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615739883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 11.723229169s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:14:18 UTC) orbiter warning: context deadline exceeded Duration: 11.723257283s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....X.......X.........................................] duration: 6.016700924s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:14:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.016731292s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 11.636787491s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:13:58 UTC) orbiter warning: context deadline exceeded Duration: 11.636816886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....X.......X........................................] duration: 5.964635564s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 5.964665782s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X........X..] duration: 5.001769827s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:13:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001807328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.....X.......X.......................................] duration: 6.919906931s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:13:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.91993825s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.921266651s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:12:51 UTC) orbiter warning: context deadline exceeded Duration: 6.921295186s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X........X.] duration: 3.831874729s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:12:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.831904285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.....X.......X......................................] duration: 6.128005008s trace: https://jaeger.dev.storj.io/trace/7c82d3b169ae7176 FAIL (2025-11-15 16:12:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ) context deadline exceeded; piecestore: (Node ID: 12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, Piece ID: JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA) context deadline exceeded ======LOGS====== 41m11.758223385s) [upload=133577]: Done starting upload Nov 15 15:45:13.206882240 (151h41m11.758352701s) [upload=133577]: Uploading segments... Nov 15 15:45:13.207060730 (151h41m11.758531191s) [upload=133577]: Got next segment. Inline: false Nov 15 15:45:13.207097951 (151h41m11.758568412s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 15 15:45:13.207104643 (151h41m11.758575104s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 15 15:45:13.476638279 (151h41m12.02810874s) [seg_pos={0 0}, upload=133577]: Begin upload segment... Nov 15 15:45:13.476698023 (151h41m12.028168484s) [seg_pos={0 0}, upload=133577]: Done begin upload segment. Nov 15 15:45:13.476720826 (151h41m12.028191287s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.476757607 (151h41m12.028228068s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477395391 (151h41m12.028865852s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.477790292 (151h41m12.029260753s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479410408 (151h41m12.030880859s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.479917724 (151h41m12.031388185s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.480909723 (151h41m12.032380184s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.481776734 (151h41m12.033247195s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.482168030 (151h41m12.033638481s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.484069950 (151h41m12.035540412s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.486878869 (151h41m12.038349331s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.489145295 (151h41m12.040615756s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.494267269 (151h41m12.04573773s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.495484577 (151h41m12.046955028s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.498573178 (151h41m12.05004363s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:13.499215531 (151h41m12.050685992s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Uploading piece... Nov 15 15:45:15.223520164 (151h41m13.774990625s) [upload=133577]: Next returned nil segment Nov 15 15:45:15.223570420 (151h41m13.775040881s) [upload=133577]: Waiting for error group managing segments... Nov 15 15:45:15.624693353 (151h41m14.176163814s) [noise=true , address=77.90.31.17:4771 , piece_id=SWCSWJAD3TY7WG52ASQPSFBMBQLGKUYIHJ3T7HRP3WAT2RMOEXKA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.629761844 (151h41m14.181232305s) [noise=true , address=77.90.31.15:4698 , piece_id=JV3TX7ZXCOX7QUGBKNYSKU5VRQ7JVO5I4U5H7IFJI7T4VYH4R2DA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.637889782 (151h41m14.189360243s) [noise=true , address=45.91.116.10:4081 , piece_id=35DSQXYB7Q47TACTKOMSCSV7RX57YXYJNS4GUFSZVWEJECJIIJMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.650497169 (151h41m14.20196763s) [noise=true , address=45.91.116.7:4010 , piece_id=RMCE6KWD4AFOA3QOLE7J6JCODU7IWHBIAEXVAWFLMHRS45M465LQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.659087527 (151h41m14.210557988s) [noise=true , address=77.90.31.13:4628 , piece_id=L3R4USWAQL5YLN46WRLEAEMCWU6BPITN3U7ZB35CWEOJUK43JIGA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.675481244 (151h41m14.226951706s) [noise=true , address=77.90.31.14:4662 , piece_id=S5QBRQJL3V4CHLH47E42TQ372LEDVXTCNOMSEEFMB7QKI57PZG6A, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:15.728692331 (151h41m14.280162782s) [noise=true , address=77.90.31.12:4589 , piece_id=DMPKETQC7WIFUDEZZQJET7POJG5MLU3YIP7ALSVZWLN4RXHPT74Q, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:18.408053417 (151h41m16.959523878s) [noise=true , address=145.79.227.21:4475 , piece_id=42XACYXGVX2EMQT6U5OK6PDMQUBEYQGASAHZTCLIGRY7MULYXNAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.822174665 (151h41m26.373645126s) [noise=true , address=145.79.227.23:4546 , piece_id=PXNAA5M4SXJ3CMNZGVINPWA5KQLWWOQMZZT7USFXL66LTVVLDXLA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:27.838206875 (151h41m26.389677336s) [noise=true , address=145.79.227.25:4619 , piece_id=UEMXRW4M77BG5DYULOUYEKRYYZDFNW6C5TE7YMDAE7RGYKFILYAA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:28.251211704 (151h41m26.802682165s) [noise=true , address=145.79.227.26:4654 , piece_id=EMY3G4JTTWCIJWY6WZUO3VKN2MK77FY5B3FYTJMSJGDSLZJP4BCQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:33.332020493 (151h41m31.883490944s) [noise=true , address=145.79.227.16:4910 , piece_id=5LAGEBTTXPFH4WDH2QEMVXLAQNADAYHJSYKCPZJ6KHG3PGE6WBAQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937270016 (151h41m38.488740477s) [noise=true , address=145.79.227.17:4971 , piece_id=YSZU5WHB4W74AOWHSKCRV5J2OZMOW64B4HC6PHFBCQ5MPKGK2IGQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: <nil> Nov 15 15:45:39.937295745 (151h41m38.488766196s) [seg_pos={0 0}, upload=133577]: Segment reached optimal threshold of 13 pieces. Nov 15 15:45:39.937493512 (151h41m38.488963973s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Nov 15 15:45:39.937504052 (151h41m38.488974523s) [noise=true , address=145.79.227.14:4838 , piece_id=JFGFU2ZT72ISC5AMESARORTRU55466CRKUDE7O63G6SIWWCPWNZA, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.937954059 (151h41m38.48942452s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37968->145.79.227.15:4875: use of closed network connection Nov 15 15:45:39.937962775 (151h41m38.489433236s) [noise=true , address=145.79.227.15:4875 , piece_id=7BHND3OZR2CY4QNDFSDM5XXNJ6NEPU5NORKE32LHQV3HRSPAXLZQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938286512 (151h41m38.489756974s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: read tcp 188.166.225.170:51304->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:45:39.938293896 (151h41m38.489764347s) [noise=true , address=145.79.227.11:4732 , piece_id=727BCKLR4JLMR4WZR43DVPGSVMWRRBCK2OUV7NJMQREHJMPSCEMQ, seg_pos={0 0}, upload=133577]: Long tail cancellation due to optimal threshold Nov 15 15:45:39.938311740 (151h41m38.489782201s) [seg_pos={0 0}, upload=133577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 15 15:45:39.938400068 (151h41m38.48987053s) [upload=133577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 15 15:45:39.938405368 (151h41m38.489875819s) [upload=133577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 15 15:45:40.255560647 (151h41m38.807031118s) [upload=133577]: Done uploading segments... Nov 15 15:45:40.255586336 (151h41m38.807056797s) [upload=133577]: Upload finished. err: <nil> Duration: 6.12803734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X........X] duration: 5.001028147s trace: https://jaeger.dev.storj.io/trace/2452be469c4d43fd FAIL (2025-11-15 16:11:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001049148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 11.941867769s trace: https://jaeger.dev.storj.io/trace/c43a36903771313 FAIL (2025-11-15 16:11:46 UTC) orbiter warning: context deadline exceeded Duration: 11.941919567s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)