Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX.XXXX...................................................] duration: 6.129771631s FAIL (2025-10-07 07:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.129798704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 5.98568246s FAIL (2025-10-07 07:30:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.985744748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XXX.XXXX..................................................] duration: 5.918344596s FAIL (2025-10-07 07:30:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.918374544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration: 6.091011988s FAIL (2025-10-07 07:30:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.091042726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.XXX.XXXX.................................................] duration: 5.872221479s FAIL (2025-10-07 07:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.872264621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXX.X......................................................] duration: 5.934245796s FAIL (2025-10-07 07:29:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.934282276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.XXX.XXXX................................................] duration: 5.739381217s FAIL (2025-10-07 07:28:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.739416784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.XXX.X.....................................................] duration: 6.211229052s FAIL (2025-10-07 07:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.211311217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.XXX.XXXX...............................................] duration: 5.673794308s FAIL (2025-10-07 07:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.673830707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.XXX.X....................................................] duration: 5.960944624s FAIL (2025-10-07 07:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.960975763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.XXX.XXXX..............................................] duration: 5.787400716s FAIL (2025-10-07 07:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.787430773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX.XXX.X...................................................] duration: 5.866662834s FAIL (2025-10-07 07:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.866692249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXX.XXX.X..................................................] duration: 6.230204857s FAIL (2025-10-07 07:26:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.230235635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X....X.XXX.XXXX.............................................] duration: 5.739708789s FAIL (2025-10-07 07:25:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.739743265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXX.XXX.X.................................................] duration: 6.22968703s FAIL (2025-10-07 07:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.229717558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X....X.XXX.XXXX............................................] duration: 6.099222109s FAIL (2025-10-07 07:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.099251415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXX.XXX.X................................................] duration: 6.358181701s FAIL (2025-10-07 07:24:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.358214703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X.XXX.XXXX...........................................] duration: 5.975798794s FAIL (2025-10-07 07:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.975862596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X....X.XXX.XXXX..........................................] duration: 6.454474883s FAIL (2025-10-07 07:23:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.454502475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXX.XXX.X...............................................] duration: 6.175776292s FAIL (2025-10-07 07:23:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.17580693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXX.XXX.X..............................................] duration: 6.976353413s FAIL (2025-10-07 07:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.976386115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXXXXXXX.XXX.X.............................................] duration: 6.20508308s FAIL (2025-10-07 07:22:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.205115243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X....X.XXX.XXXX.........................................] duration: 6.012514791s FAIL (2025-10-07 07:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.012550679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XXXXXXXX.XXX.X............................................] duration: 7.068402429s FAIL (2025-10-07 07:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 7.068432448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X....X.XXX.XXXX........................................] duration: 6.732692251s FAIL (2025-10-07 07:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.732721047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XXXXXXXX.XXX.X...........................................] duration: 6.609659163s FAIL (2025-10-07 07:21:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.609682588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X....X.XXX.XXXX.......................................] duration: 6.05568242s FAIL (2025-10-07 07:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.055706084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X....X.XXX.XXXX......................................] duration: 6.198457305s FAIL (2025-10-07 07:20:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.198486409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XXXXXXXX.XXX.X..........................................] duration: 6.197282725s FAIL (2025-10-07 07:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.19731165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X....X.XXX.XXXX.....................................] duration: 5.890984845s FAIL (2025-10-07 07:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.891014392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....XXXXXXXX.XXX.X.........................................] duration: 6.165519261s FAIL (2025-10-07 07:19:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.165546812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X....X.XXX.XXXX....................................] duration: 5.921170391s FAIL (2025-10-07 07:19:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.921267827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.....XXXXXXXX.XXX.X........................................] duration: 6.110069844s FAIL (2025-10-07 07:18:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.110106031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X....X.XXX.XXXX...................................] duration: 5.867287506s FAIL (2025-10-07 07:18:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.867324537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X....X.XXX.XXXX..................................] duration: 6.043048965s FAIL (2025-10-07 07:18:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.043076157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.....XXXXXXXX.XXX.X.......................................] duration: 6.027049178s FAIL (2025-10-07 07:18:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.027068564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X....X.XXX.XXXX.................................] duration: 5.898839242s FAIL (2025-10-07 07:17:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.898866463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.....XXXXXXXX.XXX.X......................................] duration: 6.313730134s FAIL (2025-10-07 07:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.313761312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X....X.XXX.XXXX................................] duration: 6.159459993s FAIL (2025-10-07 07:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.15948959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.....XXXXXXXX.XXX.X.....................................] duration: 6.393643716s FAIL (2025-10-07 07:16:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.393688361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.....XXXXXXXX.XXX.X....................................] duration: 6.164540512s FAIL (2025-10-07 07:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.164572364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X....X.XXX.XXXX...............................] duration: 5.994709505s FAIL (2025-10-07 07:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.994737549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.....XXXXXXXX.XXX.X...................................] duration: 6.191871226s FAIL (2025-10-07 07:15:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.191906875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X....X.XXX.XXXX..............................] duration: 5.968764073s FAIL (2025-10-07 07:15:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.968794891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.....XXXXXXXX.XXX.X..................................] duration: 6.06559478s FAIL (2025-10-07 07:14:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.065629526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X.....XXXXXXXX.XXX.X.................................] duration: 6.27429262s FAIL (2025-10-07 07:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.274364607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [................X....X.XXX.XXXX.............................] duration: 6.343621372s FAIL (2025-10-07 07:13:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.34367232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X.....XXXXXXXX.XXX.X................................] duration: 6.085747327s FAIL (2025-10-07 07:13:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.085782905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.................X....X.XXX.XXXX............................] duration: 6.03354133s FAIL (2025-10-07 07:13:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.033574001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X.....XXXXXXXX.XXX.X...............................] duration: 6.11049317s FAIL (2025-10-07 07:13:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.110531111s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)