Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX...XX] duration: 30.003918823s trace: https://jaeger.dev.storj.io/trace/6d53d39458052abb FAIL (2025-12-28 17:35:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 28 17:34:31.535881768 (554h5m31.406964813s) [upload=485790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:34:31.775387590 (554h5m31.646470635s) [seg_pos={0 0}, upload=485790]: Begin upload segment... Dec 28 17:34:31.775451492 (554h5m31.646534537s) [seg_pos={0 0}, upload=485790]: Done begin upload segment. Dec 28 17:34:31.775472962 (554h5m31.646555997s) [noise=true , address=77.90.31.12:4589 , piece_id=E5JT4XBKG3EJC5FGESR6ILT2OBQUT3VFMIU4KDGJWLJXDOWDUKCA, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.775579796 (554h5m31.646662841s) [noise=true , address=45.91.116.7:4012 , piece_id=7HX4C5SQ4UDXZ54NCRXNAMLIB4GP7MQTPF66ZRRSIG2GETPWP6MA, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.776726219 (554h5m31.647809264s) [noise=true , address=145.79.227.11:4731 , piece_id=4TWZXUR36MT5WJO2LQT6PZJNBXCM63P4VJTY2UCWFCNIASZFCV7A, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.777322113 (554h5m31.648405158s) [noise=true , address=145.79.227.45:4230 , piece_id=USDLFF46SLQCRQRPIJBPOV4FJ7DD4TM5JMVXLR2CR5IADG4MFVXQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.777752243 (554h5m31.648835288s) [noise=true , address=77.90.31.21:4878 , piece_id=YD4EYBY5FH2QLDI7KX2LF75QXMIWMDEK5VKNVEK3EIOK6XEAB45A, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.778243859 (554h5m31.649326904s) [noise=true , address=145.79.227.26:4654 , piece_id=NOHHOVVUARPIFRULDH7JCKYTWCPDM2HU7EL4ALE4Q5QW3Z6IUMPQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.778954401 (554h5m31.650037437s) [noise=true , address=145.79.227.19:4077 , piece_id=N4Y6AWHW4HYINWH4VX2MUMF5GZPEPDKUFFDW6ZRN35TFJ3YQWWJQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.779611052 (554h5m31.650694097s) [noise=true , address=45.91.116.9:4262 , piece_id=FKNIQ4KYBKZBFGWLO2TEFB2IA5CGJ57HI5XNCDOUT7UETJ4MNNMQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.780648949 (554h5m31.651731994s) [noise=true , address=145.79.227.36:4517 , piece_id=NEUOZJCI7BO2MDDRVJRCPJMF3ENXEAUN4GZHQP47XZ7UBGCTXPQQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.781175651 (554h5m31.652258696s) [noise=true , address=145.79.227.3:4151 , piece_id=AQOX7JDCGDEZBGNIDV26UAIVT4ZWJ2PNMOWUXFMJOSFLVVMP4M4A, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.781779992 (554h5m31.652863037s) [noise=true , address=145.79.227.50:4373 , piece_id=RWCMWKIF4ZOT4QVFTQ7TSPZOMELUZT7GECRUDWNDYTYADNSNDPAA, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.782645029 (554h5m31.653728074s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJRWACBUQGB5FPWID3H2D2SBV7PY42BPJQBTOMQMXQWCW7E3ODA, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.783488786 (554h5m31.654571832s) [noise=true , address=45.91.116.10:4082 , piece_id=BNPPQLA3QIWE4BZOV36YF46MKLJTY3D32MQF46IHW5IOLU47WK4Q, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.783773639 (554h5m31.654856684s) [noise=true , address=145.79.227.4:4185 , piece_id=CS5DAR3CD2EKXJW5AF2X6X34BQK6N4U6TCJVCROFNVPIB2CU7D5A, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.784392397 (554h5m31.655475452s) [noise=true , address=145.79.227.22:4510 , piece_id=4P5TV2CB2YK64SNSQSSX2BKICCAYMXKW4GWEYL6SXEXGEIMH4GKQ, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:31.785185527 (554h5m31.656268572s) [noise=true , address=77.90.31.14:4664 , piece_id=6YDMZLACGNSQA7NFIAEYITUXY75HPFH3C4DJWLGRRGAE4YV5UQYA, seg_pos={0 0}, upload=485790]: Uploading piece... Dec 28 17:34:33.483449328 (554h5m33.354532374s) [upload=485790]: Next returned nil segment Dec 28 17:34:33.483463635 (554h5m33.35454668s) [upload=485790]: Waiting for error group managing segments... Dec 28 17:34:33.776434845 (554h5m33.64751788s) [noise=true , address=145.79.227.3:4151 , piece_id=AQOX7JDCGDEZBGNIDV26UAIVT4ZWJ2PNMOWUXFMJOSFLVVMP4M4A, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.786999372 (554h5m33.658082407s) [noise=true , address=145.79.227.36:4517 , piece_id=NEUOZJCI7BO2MDDRVJRCPJMF3ENXEAUN4GZHQP47XZ7UBGCTXPQQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.787911299 (554h5m33.658994344s) [noise=true , address=145.79.227.22:4510 , piece_id=4P5TV2CB2YK64SNSQSSX2BKICCAYMXKW4GWEYL6SXEXGEIMH4GKQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.789192117 (554h5m33.660275153s) [noise=true , address=145.79.227.39:4015 , piece_id=5YJRWACBUQGB5FPWID3H2D2SBV7PY42BPJQBTOMQMXQWCW7E3ODA, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.795690943 (554h5m33.666773988s) [noise=true , address=145.79.227.45:4230 , piece_id=USDLFF46SLQCRQRPIJBPOV4FJ7DD4TM5JMVXLR2CR5IADG4MFVXQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.797443399 (554h5m33.668526434s) [noise=true , address=145.79.227.50:4373 , piece_id=RWCMWKIF4ZOT4QVFTQ7TSPZOMELUZT7GECRUDWNDYTYADNSNDPAA, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.806948479 (554h5m33.678031524s) [noise=true , address=145.79.227.4:4185 , piece_id=CS5DAR3CD2EKXJW5AF2X6X34BQK6N4U6TCJVCROFNVPIB2CU7D5A, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.815652924 (554h5m33.686735959s) [noise=true , address=145.79.227.11:4731 , piece_id=4TWZXUR36MT5WJO2LQT6PZJNBXCM63P4VJTY2UCWFCNIASZFCV7A, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.939257489 (554h5m33.810340534s) [noise=true , address=145.79.227.19:4077 , piece_id=N4Y6AWHW4HYINWH4VX2MUMF5GZPEPDKUFFDW6ZRN35TFJ3YQWWJQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:33.960582526 (554h5m33.831665561s) [noise=true , address=77.90.31.12:4589 , piece_id=E5JT4XBKG3EJC5FGESR6ILT2OBQUT3VFMIU4KDGJWLJXDOWDUKCA, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:34.111578079 (554h5m33.982661114s) [noise=true , address=45.91.116.7:4012 , piece_id=7HX4C5SQ4UDXZ54NCRXNAMLIB4GP7MQTPF66ZRRSIG2GETPWP6MA, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:34.123659965 (554h5m33.994743s) [noise=true , address=45.91.116.9:4262 , piece_id=FKNIQ4KYBKZBFGWLO2TEFB2IA5CGJ57HI5XNCDOUT7UETJ4MNNMQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:34.162316614 (554h5m34.033399659s) [noise=true , address=45.91.116.10:4082 , piece_id=BNPPQLA3QIWE4BZOV36YF46MKLJTY3D32MQF46IHW5IOLU47WK4Q, seg_pos={0 0}, upload=485790]: Done uploading piece. err: <nil> Dec 28 17:34:34.162335500 (554h5m34.033418535s) [seg_pos={0 0}, upload=485790]: Segment reached optimal threshold of 13 pieces. Dec 28 17:34:34.162568975 (554h5m34.03365202s) [noise=true , address=145.79.227.26:4654 , piece_id=NOHHOVVUARPIFRULDH7JCKYTWCPDM2HU7EL4ALE4Q5QW3Z6IUMPQ, seg_pos={0 0}, upload=485790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:47470->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:47470->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47470->145.79.227.26:4654: use of closed network connection Dec 28 17:34:34.162575617 (554h5m34.033658652s) [noise=true , address=145.79.227.26:4654 , piece_id=NOHHOVVUARPIFRULDH7JCKYTWCPDM2HU7EL4ALE4Q5QW3Z6IUMPQ, seg_pos={0 0}, upload=485790]: Long tail cancellation due to optimal threshold Dec 28 17:34:34.162590936 (554h5m34.033673971s) [noise=true , address=77.90.31.21:4878 , piece_id=YD4EYBY5FH2QLDI7KX2LF75QXMIWMDEK5VKNVEK3EIOK6XEAB45A, seg_pos={0 0}, upload=485790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:54104->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:54104->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54104->77.90.31.21:4878: use of closed network connection Dec 28 17:34:34.162600354 (554h5m34.033683389s) [noise=true , address=77.90.31.21:4878 , piece_id=YD4EYBY5FH2QLDI7KX2LF75QXMIWMDEK5VKNVEK3EIOK6XEAB45A, seg_pos={0 0}, upload=485790]: Long tail cancellation due to optimal threshold Dec 28 17:34:34.162723969 (554h5m34.033807004s) [noise=true , address=77.90.31.14:4664 , piece_id=6YDMZLACGNSQA7NFIAEYITUXY75HPFH3C4DJWLGRRGAE4YV5UQYA, seg_pos={0 0}, upload=485790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:41546->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:41546->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41546->77.90.31.14:4664: use of closed network connection Dec 28 17:34:34.162732015 (554h5m34.03381506s) [noise=true , address=77.90.31.14:4664 , piece_id=6YDMZLACGNSQA7NFIAEYITUXY75HPFH3C4DJWLGRRGAE4YV5UQYA, seg_pos={0 0}, upload=485790]: Long tail cancellation due to optimal threshold Dec 28 17:34:34.162800345 (554h5m34.033883391s) [seg_pos={0 0}, upload=485790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:34:34.162944700 (554h5m34.034027735s) [upload=485790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:34:34.162952014 (554h5m34.034035059s) [upload=485790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:34:34.445959431 (554h5m34.317042476s) [upload=485790]: Done uploading segments... Dec 28 17:34:34.445980532 (554h5m34.317063577s) [upload=485790]: Upload finished. err: <nil> Duration:* 30.003931908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 45.122380452s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:34:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 45.122407983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX...X] duration: 30.003588328s trace: https://jaeger.dev.storj.io/trace/25c457711aec79e FAIL (2025-12-28 17:34:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: 6EFLUX33HWUACN6QKPCL5QQCUYRYID5XAVBJAYPKO65HEIHPNQIQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: FGMASO32NWVSEYCZKNQZJIMPGP32T4UGRUR3DVVKB2GY4NX3K7BQ) context deadline exceeded ======LOGS====== Dec 28 17:34:01.282597243 (554h5m1.153680288s) [upload=485783]: Starting upload Dec 28 17:34:01.282668028 (554h5m1.153751073s) [upload=485783]: Done starting upload Dec 28 17:34:01.282789590 (554h5m1.153872655s) [upload=485783]: Uploading segments... Dec 28 17:34:01.283020669 (554h5m1.154103714s) [upload=485783]: Got next segment. Inline: false Dec 28 17:34:01.283071907 (554h5m1.154154952s) [upload=485783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:34:01.283108266 (554h5m1.154191311s) [upload=485783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:34:01.532279704 (554h5m1.403362759s) [seg_pos={0 0}, upload=485783]: Begin upload segment... Dec 28 17:34:01.532332444 (554h5m1.403415489s) [seg_pos={0 0}, upload=485783]: Done begin upload segment. Dec 28 17:34:01.532366569 (554h5m1.403449614s) [noise=true , address=145.79.227.26:4654 , piece_id=7BDWIMJZXTRS3ZIADY5LRV6T54WTQQO4UNF55EQL7H77ITE3OTXQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.534217874 (554h5m1.405300949s) [noise=true , address=77.90.31.13:4629 , piece_id=6AUETVBTWOHKYPAVHR7ISLYNQDHA7FIXKNWDHB5HHCZOB55TQX6A, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.535312930 (554h5m1.406395975s) [noise=true , address=145.79.227.51:4411 , piece_id=73TSMJVSJHYJBS74U5VOWYC6EMTZHAAZLLCAQKSNYCBGGNP7JKGQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.536418355 (554h5m1.4075014s) [noise=true , address=145.79.227.11:4731 , piece_id=F2RQTXYQWK3K6XV5PLFKSEI5BZ3RZ4ROWJMUHMWXJ27QVMPFUGCQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.537630091 (554h5m1.408713136s) [noise=true , address=145.79.227.15:4873 , piece_id=6IGOCC2GEKUIOVYTRTGGIG474Z5BZQH7JW5NJWLNT5ZGGUXZ2R7A, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.538397933 (554h5m1.409480978s) [noise=true , address=77.90.31.16:4735 , piece_id=VTDKXZHP2FSIOZKOGSGDKYFYOR52PVEGYNJTYSMW65A5I52ELK3A, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.539013335 (554h5m1.410096381s) [noise=true , address=145.79.227.50:4375 , piece_id=5ASUV5SAOLKC2B73KCBSTZRBSTNIU2OIZVCC5TBNSFVEOXFKCHUA, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.539506274 (554h5m1.410589329s) [noise=true , address=45.91.116.11:4118 , piece_id=GXYKMXZ7RXI5H5PJ2F3PQ6JSPQBEPWWU2ELOOBGFLFEMGVFIAI6A, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.540357245 (554h5m1.41144031s) [noise=true , address=145.79.227.40:4051 , piece_id=DXLEJEBSGDM2NNFD7XYKEIZETOLIUCN4EOXL6FMMMVMA72YSF7KQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.541245666 (554h5m1.412328741s) [noise=true , address=145.79.227.36:4518 , piece_id=RXDKESSEIAMMONACDGT6R4CLQXH7ZFG3HD2GNM6V6VYH3XOT2B3Q, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.542694636 (554h5m1.413777691s) [noise=true , address=45.91.116.10:4082 , piece_id=4BRAYZDFF5Q6GCSWZYMBKYRVOWJC3EHIWMNE47GLCKUYR4YP3GPQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.543381975 (554h5m1.41446503s) [noise=true , address=145.79.227.46:4602 , piece_id=LPTJZP3EE7CB4HRK2KFOGFHKXE6TGMWPUZIFGZXHO224MOJFQTRQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.544179764 (554h5m1.415262809s) [noise=true , address=45.91.116.6:4974 , piece_id=FGMASO32NWVSEYCZKNQZJIMPGP32T4UGRUR3DVVKB2GY4NX3K7BQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.545002490 (554h5m1.416085545s) [noise=true , address=145.79.227.32:4835 , piece_id=G56UQOML2MBWGDQ7JOKEOWTIRALUPFZITU262HJCLMYSJ4T236GQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.546134897 (554h5m1.417217942s) [noise=true , address=145.79.227.3:4151 , piece_id=OUV5H3EWBXQ5ZDVZWYO5SDR4XQ7B6PCMSPARACS4QQQ6TJBYNLUQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:01.546761199 (554h5m1.417844234s) [noise=true , address=77.90.31.11:4555 , piece_id=6EFLUX33HWUACN6QKPCL5QQCUYRYID5XAVBJAYPKO65HEIHPNQIQ, seg_pos={0 0}, upload=485783]: Uploading piece... Dec 28 17:34:03.226497010 (554h5m3.097580055s) [upload=485783]: Next returned nil segment Dec 28 17:34:03.226527348 (554h5m3.097610403s) [upload=485783]: Waiting for error group managing segments... Dec 28 17:34:03.548780957 (554h5m3.419864002s) [noise=true , address=145.79.227.50:4375 , piece_id=5ASUV5SAOLKC2B73KCBSTZRBSTNIU2OIZVCC5TBNSFVEOXFKCHUA, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.551139207 (554h5m3.422222252s) [noise=true , address=145.79.227.11:4731 , piece_id=F2RQTXYQWK3K6XV5PLFKSEI5BZ3RZ4ROWJMUHMWXJ27QVMPFUGCQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.551829571 (554h5m3.422912636s) [noise=true , address=145.79.227.51:4411 , piece_id=73TSMJVSJHYJBS74U5VOWYC6EMTZHAAZLLCAQKSNYCBGGNP7JKGQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.554416297 (554h5m3.425499342s) [noise=true , address=145.79.227.36:4518 , piece_id=RXDKESSEIAMMONACDGT6R4CLQXH7ZFG3HD2GNM6V6VYH3XOT2B3Q, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.566538640 (554h5m3.437621675s) [noise=true , address=145.79.227.15:4873 , piece_id=6IGOCC2GEKUIOVYTRTGGIG474Z5BZQH7JW5NJWLNT5ZGGUXZ2R7A, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.567582678 (554h5m3.438665723s) [noise=true , address=145.79.227.46:4602 , piece_id=LPTJZP3EE7CB4HRK2KFOGFHKXE6TGMWPUZIFGZXHO224MOJFQTRQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.573645493 (554h5m3.444728538s) [noise=true , address=145.79.227.26:4654 , piece_id=7BDWIMJZXTRS3ZIADY5LRV6T54WTQQO4UNF55EQL7H77ITE3OTXQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.574142610 (554h5m3.445225645s) [noise=true , address=145.79.227.32:4835 , piece_id=G56UQOML2MBWGDQ7JOKEOWTIRALUPFZITU262HJCLMYSJ4T236GQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.795143505 (554h5m3.66622655s) [noise=true , address=77.90.31.16:4735 , piece_id=VTDKXZHP2FSIOZKOGSGDKYFYOR52PVEGYNJTYSMW65A5I52ELK3A, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.806515119 (554h5m3.677598164s) [noise=true , address=77.90.31.11:4555 , piece_id=6EFLUX33HWUACN6QKPCL5QQCUYRYID5XAVBJAYPKO65HEIHPNQIQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.896562706 (554h5m3.767645741s) [noise=true , address=45.91.116.10:4082 , piece_id=4BRAYZDFF5Q6GCSWZYMBKYRVOWJC3EHIWMNE47GLCKUYR4YP3GPQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:03.915835134 (554h5m3.786918179s) [noise=true , address=45.91.116.6:4974 , piece_id=FGMASO32NWVSEYCZKNQZJIMPGP32T4UGRUR3DVVKB2GY4NX3K7BQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:04.925064719 (554h5m4.796147765s) [noise=true , address=145.79.227.40:4051 , piece_id=DXLEJEBSGDM2NNFD7XYKEIZETOLIUCN4EOXL6FMMMVMA72YSF7KQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: <nil> Dec 28 17:34:04.925096640 (554h5m4.796179685s) [seg_pos={0 0}, upload=485783]: Segment reached optimal threshold of 13 pieces. Dec 28 17:34:04.925347468 (554h5m4.796430513s) [noise=true , address=145.79.227.3:4151 , piece_id=OUV5H3EWBXQ5ZDVZWYO5SDR4XQ7B6PCMSPARACS4QQQ6TJBYNLUQ, seg_pos={0 0}, upload=485783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 28 17:34:04.925356996 (554h5m4.796440031s) [noise=true , address=145.79.227.3:4151 , piece_id=OUV5H3EWBXQ5ZDVZWYO5SDR4XQ7B6PCMSPARACS4QQQ6TJBYNLUQ, seg_pos={0 0}, upload=485783]: Long tail cancellation due to optimal threshold Dec 28 17:34:04.925645195 (554h5m4.79672824s) [noise=true , address=45.91.116.11:4118 , piece_id=GXYKMXZ7RXI5H5PJ2F3PQ6JSPQBEPWWU2ELOOBGFLFEMGVFIAI6A, seg_pos={0 0}, upload=485783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:34:04.925655314 (554h5m4.796738349s) [noise=true , address=45.91.116.11:4118 , piece_id=GXYKMXZ7RXI5H5PJ2F3PQ6JSPQBEPWWU2ELOOBGFLFEMGVFIAI6A, seg_pos={0 0}, upload=485783]: Long tail cancellation due to optimal threshold Dec 28 17:34:04.925672767 (554h5m4.796755812s) [noise=true , address=77.90.31.13:4629 , piece_id=6AUETVBTWOHKYPAVHR7ISLYNQDHA7FIXKNWDHB5HHCZOB55TQX6A, seg_pos={0 0}, upload=485783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 17:34:04.925700790 (554h5m4.796783866s) [noise=true , address=77.90.31.13:4629 , piece_id=6AUETVBTWOHKYPAVHR7ISLYNQDHA7FIXKNWDHB5HHCZOB55TQX6A, seg_pos={0 0}, upload=485783]: Long tail cancellation due to optimal threshold Dec 28 17:34:04.925747931 (554h5m4.796830976s) [seg_pos={0 0}, upload=485783]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:34:04.926044725 (554h5m4.797127781s) [upload=485783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:34:04.926055255 (554h5m4.79713829s) [upload=485783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:34:05.207613825 (554h5m5.07869687s) [upload=485783]: Done uploading segments... Dec 28 17:34:05.207637800 (554h5m5.078720835s) [upload=485783]: Upload finished. err: <nil> Duration: 30.003611222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...XXX.X.......X..X....................X..X.............X...] duration: 3.705895994s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:34:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.705920523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX...] duration: 6.050577064s trace: https://jaeger.dev.storj.io/trace/37a5c2573186a8ea FAIL (2025-12-28 17:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A) context deadline exceeded ======LOGS====== Dec 28 17:30:51.566537945 (554h1m51.43762098s) [upload=485738]: Starting upload Dec 28 17:30:51.566602158 (554h1m51.437685193s) [upload=485738]: Done starting upload Dec 28 17:30:51.566632836 (554h1m51.437715871s) [upload=485738]: Uploading segments... Dec 28 17:30:51.566784184 (554h1m51.437867219s) [upload=485738]: Got next segment. Inline: false Dec 28 17:30:51.566828067 (554h1m51.437911103s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:30:51.566833028 (554h1m51.437916063s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:30:51.807006501 (554h1m51.678089536s) [seg_pos={0 0}, upload=485738]: Begin upload segment... Dec 28 17:30:51.807048551 (554h1m51.678131586s) [seg_pos={0 0}, upload=485738]: Done begin upload segment. Dec 28 17:30:51.807074641 (554h1m51.678157686s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807097544 (554h1m51.678180579s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807487105 (554h1m51.678570141s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807799490 (554h1m51.678882535s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808164346 (554h1m51.679247391s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808430743 (554h1m51.679513778s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808698883 (554h1m51.679781918s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808975690 (554h1m51.680058725s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.809313454 (554h1m51.680396489s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811047175 (554h1m51.68213021s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811085528 (554h1m51.682168573s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.812791386 (554h1m51.683874421s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.814464392 (554h1m51.685547427s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.816023281 (554h1m51.687106316s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.817583762 (554h1m51.688666807s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.819351087 (554h1m51.690434122s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:53.419117512 (554h1m53.290200557s) [upload=485738]: Next returned nil segment Dec 28 17:30:53.419133031 (554h1m53.290216066s) [upload=485738]: Waiting for error group managing segments... Dec 28 17:30:53.809120068 (554h1m53.680203103s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.828506793 (554h1m53.699589839s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.830414196 (554h1m53.701497241s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.838157710 (554h1m53.709240745s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.849213463 (554h1m53.720296498s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.863382944 (554h1m53.734465989s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.877683825 (554h1m53.74876686s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.878672659 (554h1m53.749755704s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.017232982 (554h1m53.888316017s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.045252877 (554h1m53.916335912s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.107542838 (554h1m53.978625884s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.375128871 (554h1m54.246211906s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725460276 (554h1m54.596543311s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725483920 (554h1m54.596566966s) [seg_pos={0 0}, upload=485738]: Segment reached optimal threshold of 13 pieces. Dec 28 17:30:54.725716654 (554h1m54.596799699s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.725724859 (554h1m54.596807894s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726126675 (554h1m54.59720971s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.726138457 (554h1m54.597221532s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726334541 (554h1m54.597417576s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection Dec 28 17:30:54.726345872 (554h1m54.597428917s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726377683 (554h1m54.597460728s) [seg_pos={0 0}, upload=485738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:30:54.726497782 (554h1m54.597580837s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:30:54.726507019 (554h1m54.597590064s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:30:55.028805259 (554h1m54.899888334s) [upload=485738]: Done uploading segments... Dec 28 17:30:55.028829045 (554h1m54.89991208s) [upload=485738]: Upload finished. err: <nil> Duration: 6.050606149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 41.603128103s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:33:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 41.60315851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....XXX.X.......X..X....................X..X.............X..] duration: 3.605076508s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:33:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605106343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX..] duration: 5.68892968s trace: https://jaeger.dev.storj.io/trace/37a5c2573186a8ea FAIL (2025-12-28 17:32:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A) context deadline exceeded ======LOGS====== Dec 28 17:30:51.566537945 (554h1m51.43762098s) [upload=485738]: Starting upload Dec 28 17:30:51.566602158 (554h1m51.437685193s) [upload=485738]: Done starting upload Dec 28 17:30:51.566632836 (554h1m51.437715871s) [upload=485738]: Uploading segments... Dec 28 17:30:51.566784184 (554h1m51.437867219s) [upload=485738]: Got next segment. Inline: false Dec 28 17:30:51.566828067 (554h1m51.437911103s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:30:51.566833028 (554h1m51.437916063s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:30:51.807006501 (554h1m51.678089536s) [seg_pos={0 0}, upload=485738]: Begin upload segment... Dec 28 17:30:51.807048551 (554h1m51.678131586s) [seg_pos={0 0}, upload=485738]: Done begin upload segment. Dec 28 17:30:51.807074641 (554h1m51.678157686s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807097544 (554h1m51.678180579s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807487105 (554h1m51.678570141s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807799490 (554h1m51.678882535s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808164346 (554h1m51.679247391s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808430743 (554h1m51.679513778s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808698883 (554h1m51.679781918s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808975690 (554h1m51.680058725s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.809313454 (554h1m51.680396489s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811047175 (554h1m51.68213021s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811085528 (554h1m51.682168573s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.812791386 (554h1m51.683874421s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.814464392 (554h1m51.685547427s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.816023281 (554h1m51.687106316s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.817583762 (554h1m51.688666807s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.819351087 (554h1m51.690434122s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:53.419117512 (554h1m53.290200557s) [upload=485738]: Next returned nil segment Dec 28 17:30:53.419133031 (554h1m53.290216066s) [upload=485738]: Waiting for error group managing segments... Dec 28 17:30:53.809120068 (554h1m53.680203103s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.828506793 (554h1m53.699589839s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.830414196 (554h1m53.701497241s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.838157710 (554h1m53.709240745s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.849213463 (554h1m53.720296498s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.863382944 (554h1m53.734465989s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.877683825 (554h1m53.74876686s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.878672659 (554h1m53.749755704s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.017232982 (554h1m53.888316017s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.045252877 (554h1m53.916335912s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.107542838 (554h1m53.978625884s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.375128871 (554h1m54.246211906s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725460276 (554h1m54.596543311s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725483920 (554h1m54.596566966s) [seg_pos={0 0}, upload=485738]: Segment reached optimal threshold of 13 pieces. Dec 28 17:30:54.725716654 (554h1m54.596799699s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.725724859 (554h1m54.596807894s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726126675 (554h1m54.59720971s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.726138457 (554h1m54.597221532s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726334541 (554h1m54.597417576s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection Dec 28 17:30:54.726345872 (554h1m54.597428917s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726377683 (554h1m54.597460728s) [seg_pos={0 0}, upload=485738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:30:54.726497782 (554h1m54.597580837s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:30:54.726507019 (554h1m54.597590064s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:30:55.028805259 (554h1m54.899888334s) [upload=485738]: Done uploading segments... Dec 28 17:30:55.028829045 (554h1m54.89991208s) [upload=485738]: Upload finished. err: <nil> Duration: 5.688958714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....XXX.X.......X..X....................X..X.............X.] duration: 3.483615803s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:32:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483645722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 36.899735848s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:32:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 36.899764132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX.] duration: 6.020765637s trace: https://jaeger.dev.storj.io/trace/37a5c2573186a8ea FAIL (2025-12-28 17:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A) context deadline exceeded ======LOGS====== Dec 28 17:30:51.566537945 (554h1m51.43762098s) [upload=485738]: Starting upload Dec 28 17:30:51.566602158 (554h1m51.437685193s) [upload=485738]: Done starting upload Dec 28 17:30:51.566632836 (554h1m51.437715871s) [upload=485738]: Uploading segments... Dec 28 17:30:51.566784184 (554h1m51.437867219s) [upload=485738]: Got next segment. Inline: false Dec 28 17:30:51.566828067 (554h1m51.437911103s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:30:51.566833028 (554h1m51.437916063s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:30:51.807006501 (554h1m51.678089536s) [seg_pos={0 0}, upload=485738]: Begin upload segment... Dec 28 17:30:51.807048551 (554h1m51.678131586s) [seg_pos={0 0}, upload=485738]: Done begin upload segment. Dec 28 17:30:51.807074641 (554h1m51.678157686s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807097544 (554h1m51.678180579s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807487105 (554h1m51.678570141s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807799490 (554h1m51.678882535s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808164346 (554h1m51.679247391s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808430743 (554h1m51.679513778s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808698883 (554h1m51.679781918s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808975690 (554h1m51.680058725s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.809313454 (554h1m51.680396489s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811047175 (554h1m51.68213021s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811085528 (554h1m51.682168573s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.812791386 (554h1m51.683874421s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.814464392 (554h1m51.685547427s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.816023281 (554h1m51.687106316s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.817583762 (554h1m51.688666807s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.819351087 (554h1m51.690434122s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:53.419117512 (554h1m53.290200557s) [upload=485738]: Next returned nil segment Dec 28 17:30:53.419133031 (554h1m53.290216066s) [upload=485738]: Waiting for error group managing segments... Dec 28 17:30:53.809120068 (554h1m53.680203103s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.828506793 (554h1m53.699589839s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.830414196 (554h1m53.701497241s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.838157710 (554h1m53.709240745s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.849213463 (554h1m53.720296498s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.863382944 (554h1m53.734465989s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.877683825 (554h1m53.74876686s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.878672659 (554h1m53.749755704s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.017232982 (554h1m53.888316017s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.045252877 (554h1m53.916335912s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.107542838 (554h1m53.978625884s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.375128871 (554h1m54.246211906s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725460276 (554h1m54.596543311s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725483920 (554h1m54.596566966s) [seg_pos={0 0}, upload=485738]: Segment reached optimal threshold of 13 pieces. Dec 28 17:30:54.725716654 (554h1m54.596799699s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.725724859 (554h1m54.596807894s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726126675 (554h1m54.59720971s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.726138457 (554h1m54.597221532s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726334541 (554h1m54.597417576s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection Dec 28 17:30:54.726345872 (554h1m54.597428917s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726377683 (554h1m54.597460728s) [seg_pos={0 0}, upload=485738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:30:54.726497782 (554h1m54.597580837s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:30:54.726507019 (554h1m54.597590064s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:30:55.028805259 (554h1m54.899888334s) [upload=485738]: Done uploading segments... Dec 28 17:30:55.028829045 (554h1m54.89991208s) [upload=485738]: Upload finished. err: <nil> Duration: 6.020793489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.....XXX.X.......X..X....................X..X.............X] duration: 5.001165022s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 17:31:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001175512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 40.501644555s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:31:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 40.501658841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...XX] duration: 30.003099229s trace: https://jaeger.dev.storj.io/trace/37a5c2573186a8ea FAIL (2025-12-28 17:31:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A) context deadline exceeded ======LOGS====== Dec 28 17:30:51.566537945 (554h1m51.43762098s) [upload=485738]: Starting upload Dec 28 17:30:51.566602158 (554h1m51.437685193s) [upload=485738]: Done starting upload Dec 28 17:30:51.566632836 (554h1m51.437715871s) [upload=485738]: Uploading segments... Dec 28 17:30:51.566784184 (554h1m51.437867219s) [upload=485738]: Got next segment. Inline: false Dec 28 17:30:51.566828067 (554h1m51.437911103s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:30:51.566833028 (554h1m51.437916063s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:30:51.807006501 (554h1m51.678089536s) [seg_pos={0 0}, upload=485738]: Begin upload segment... Dec 28 17:30:51.807048551 (554h1m51.678131586s) [seg_pos={0 0}, upload=485738]: Done begin upload segment. Dec 28 17:30:51.807074641 (554h1m51.678157686s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807097544 (554h1m51.678180579s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807487105 (554h1m51.678570141s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.807799490 (554h1m51.678882535s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808164346 (554h1m51.679247391s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808430743 (554h1m51.679513778s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808698883 (554h1m51.679781918s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.808975690 (554h1m51.680058725s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.809313454 (554h1m51.680396489s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811047175 (554h1m51.68213021s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.811085528 (554h1m51.682168573s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.812791386 (554h1m51.683874421s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.814464392 (554h1m51.685547427s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.816023281 (554h1m51.687106316s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.817583762 (554h1m51.688666807s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:51.819351087 (554h1m51.690434122s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Uploading piece... Dec 28 17:30:53.419117512 (554h1m53.290200557s) [upload=485738]: Next returned nil segment Dec 28 17:30:53.419133031 (554h1m53.290216066s) [upload=485738]: Waiting for error group managing segments... Dec 28 17:30:53.809120068 (554h1m53.680203103s) [noise=true , address=145.79.227.15:4876 , piece_id=MS23MMFPORL7IK2M33LNTJLGZPEGRDXMBUEHQH3KDOMHHRCGY6OQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.828506793 (554h1m53.699589839s) [noise=true , address=145.79.227.41:4087 , piece_id=P2CJGKQUSDK33PKNAXS3MXJMFOHNTSX5PS6N7YL3H5JGTLTRWCHA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.830414196 (554h1m53.701497241s) [noise=true , address=145.79.227.14:4840 , piece_id=TV4PXHKKXCUG6FT4DQM3UPJBLEAGUHFVVV7R4QKSQIU4XEKMPV2A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.838157710 (554h1m53.709240745s) [noise=true , address=145.79.227.37:4566 , piece_id=6A2OOC54IB55R367WT2Y5XXZUOYORJFYFKVN265CY4VCJ7V5FD4A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.849213463 (554h1m53.720296498s) [noise=true , address=145.79.227.38:4978 , piece_id=ZSL2KI7R4E527TXGTA54J6UI6YLRMMFU43HUV4CFJ27634KN4NJQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.863382944 (554h1m53.734465989s) [noise=true , address=145.79.227.47:4265 , piece_id=MFHXGOKM6YBPA766PIY2U76TCPO23G2SNKVYYT7RIXSK453XF3DA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.877683825 (554h1m53.74876686s) [noise=true , address=145.79.227.48:4303 , piece_id=J3CXNC4YQH7Z346DQTOLLYUUUN4KFUFWZ7UEERXQYMYXR47PYAXQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:53.878672659 (554h1m53.749755704s) [noise=true , address=145.79.227.42:4122 , piece_id=EG3BRGMBF5GDNVID6HQPD54D7EALUNZWAAPENI6J6OZE3VR7DEUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.017232982 (554h1m53.888316017s) [noise=true , address=45.91.116.9:4262 , piece_id=W2MDBVOPYRJWXAPUSB32ARN5GKSBIC577JI6OST5IWLE5KZEMWUA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.045252877 (554h1m53.916335912s) [noise=true , address=45.91.116.5:4298 , piece_id=2ABPB423PELROWVCL62VWW52J7ADMUMPTOW7BDCNESEHTKXUGINA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.107542838 (554h1m53.978625884s) [noise=true , address=45.91.116.14:4191 , piece_id=WLOJYUWEJ7SFTQN6P3CY2Q5AL3VZIWHDPEARCKFBZCKN6HO4TLZQ, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.375128871 (554h1m54.246211906s) [noise=true , address=77.90.31.18:4808 , piece_id=V3M2XJDWFHNE5XBMES5F6NWLZPGT7XDSHZ5RVYKBP5ZJLSC2RU3Q, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725460276 (554h1m54.596543311s) [noise=true , address=77.90.31.14:4664 , piece_id=CQLZC5TOWPCUL5UYNR6YUZ4AWSK4GSWJ6N7ZUCFV6TOXENSBAS7A, seg_pos={0 0}, upload=485738]: Done uploading piece. err: <nil> Dec 28 17:30:54.725483920 (554h1m54.596566966s) [seg_pos={0 0}, upload=485738]: Segment reached optimal threshold of 13 pieces. Dec 28 17:30:54.725716654 (554h1m54.596799699s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.725724859 (554h1m54.596807894s) [noise=true , address=145.79.227.29:4941 , piece_id=ADVI4O42EYMCNWZKZ5JCCHFHJ6GL6ISCTKQW4H2RYELI7A5R64IA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726126675 (554h1m54.59720971s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:54.726138457 (554h1m54.597221532s) [noise=true , address=145.79.227.35:4481 , piece_id=FUUYAYTCTGCRXNZZBPHBNWA3JVCRE5PODOI3FIY7F3HDIWZDBSNA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726334541 (554h1m54.597417576s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33974->77.90.31.10:4517: use of closed network connection Dec 28 17:30:54.726345872 (554h1m54.597428917s) [noise=true , address=77.90.31.10:4517 , piece_id=DHBD6NROACHNH7SUW7U4YNHIPZKEU3WJPZW6VG33NUFZBEHGQIFA, seg_pos={0 0}, upload=485738]: Long tail cancellation due to optimal threshold Dec 28 17:30:54.726377683 (554h1m54.597460728s) [seg_pos={0 0}, upload=485738]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:30:54.726497782 (554h1m54.597580837s) [upload=485738]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:30:54.726507019 (554h1m54.597590064s) [upload=485738]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:30:55.028805259 (554h1m54.899888334s) [upload=485738]: Done uploading segments... Dec 28 17:30:55.028829045 (554h1m54.89991208s) [upload=485738]: Upload finished. err: <nil> Duration: 30.003122042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 45.126452235s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:30:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 45.126482242s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.....XXX.X.......X..X....................X..X.............] duration: 3.495866457s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:30:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495889411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...X] duration: 30.003418632s trace: https://jaeger.dev.storj.io/trace/30b6f943ff56d7be FAIL (2025-12-28 17:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: XWQMNUGXN3VQYDTZOOTNHVQZ5AVIAW6TFWQTXJ4PU2GIYQMAPCYA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: PEJR4LHCWENG6LTUMUVBW5F5XMKSHRKGSRQWJ4TVBLKAUPKBXGPQ) context deadline exceeded ======LOGS====== Dec 28 17:30:05.264141411 (554h1m5.135224456s) [upload=485728]: Starting upload Dec 28 17:30:05.264225972 (554h1m5.135309017s) [upload=485728]: Done starting upload Dec 28 17:30:05.264492780 (554h1m5.135575825s) [upload=485728]: Uploading segments... Dec 28 17:30:05.264538217 (554h1m5.135621282s) [upload=485728]: Got next segment. Inline: false Dec 28 17:30:05.264570879 (554h1m5.135653914s) [upload=485728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:30:05.264576730 (554h1m5.135659775s) [upload=485728]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:30:05.510928928 (554h1m5.382011964s) [seg_pos={0 0}, upload=485728]: Begin upload segment... Dec 28 17:30:05.511022377 (554h1m5.382105422s) [seg_pos={0 0}, upload=485728]: Done begin upload segment. Dec 28 17:30:05.511087110 (554h1m5.382170165s) [noise=true , address=77.90.31.18:4808 , piece_id=35MESIN2JGX2LH6U4BKMH7XNIYKZODPKL6RRRSWTZYM2ZI5IQO5A, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.511097810 (554h1m5.382180855s) [noise=true , address=145.79.227.37:4567 , piece_id=AN56FQIJVZHIEIIYOSQJEKPMDMXKLR4X6OBCOGKJK3DTHZ3LONBQ, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.511176289 (554h1m5.382259324s) [noise=true , address=45.91.116.4:4939 , piece_id=LKH4Y3D7GFBBBUU7TFLHM7TUCK7477ZUC6JG2TJT5D32B5VG7O7Q, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.511691270 (554h1m5.382774315s) [noise=true , address=45.91.116.5:4297 , piece_id=32ZEP2ET5EZXQWJ6TWQZBF3RZ2MOKJRYQAAPFNAVKANDBI5TFK5A, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.512089840 (554h1m5.383172885s) [noise=true , address=45.91.116.7:4011 , piece_id=3RCTIMF3U7I65PEIF524MWYPARHOK4RVGBTMPHKBD43FLGAQRTCA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.512584171 (554h1m5.383667216s) [noise=true , address=145.79.227.2:4113 , piece_id=ED5WPPMQMSBKO5QXALSTXHKPZX3OYF6QDGRMOUUU2KZB5FCI35FA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.512973933 (554h1m5.384056988s) [noise=true , address=145.79.227.11:4729 , piece_id=PBEHUH44SSE2II72U27PVUQ3DUTWA3FWX35KCSDFBXD7LVB6CVWA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.513348296 (554h1m5.384431331s) [noise=true , address=45.91.116.14:4190 , piece_id=XWQMNUGXN3VQYDTZOOTNHVQZ5AVIAW6TFWQTXJ4PU2GIYQMAPCYA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.513884376 (554h1m5.384967431s) [noise=true , address=145.79.227.4:4187 , piece_id=QI66LMP2CYC5R2DYJ5UPB7T4ALYGNZABZI2IEOWNWBOWSPXFIWVQ, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.514661155 (554h1m5.3857442s) [noise=true , address=77.90.31.16:4736 , piece_id=PEJR4LHCWENG6LTUMUVBW5F5XMKSHRKGSRQWJ4TVBLKAUPKBXGPQ, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.516533210 (554h1m5.387616285s) [noise=true , address=145.79.227.41:4085 , piece_id=436TDYTOAQQY2SISNNUQJPV7V4DWGFWMLSVIH7YVZ7FDVCI42THA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.517324437 (554h1m5.388407482s) [noise=true , address=77.90.31.14:4664 , piece_id=2C6FZTSSUGD5U7B2VUXLULA2ADXLFVLRC6GG5W5M7ZZWUW6PNZMQ, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.517861369 (554h1m5.388944414s) [noise=true , address=145.79.227.36:4518 , piece_id=5BGPNQ2N7GWWYKGZZTWIGHNFL5TBQVI6OXWFBE5VGLENIXL2CM5Q, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.518381830 (554h1m5.389464875s) [noise=true , address=145.79.227.3:4149 , piece_id=JGV2WF4WLQOMDTAEM6OPZZMLEXUSAVZYBXZGOMKRKOVFPBM66QSQ, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.519732873 (554h1m5.390815918s) [noise=true , address=77.90.31.11:4553 , piece_id=MMNX6C2HKQ3Z3TWQEET6DBSBEFME3AV7UQLDSGP6ZTCMXHKUEPBA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:05.520661732 (554h1m5.391744767s) [noise=true , address=45.91.116.11:4120 , piece_id=EOM5TZNJ4M4OAAOZ77F4IESINYF3JDRMN25LWS4TJJVCWQOJBJEA, seg_pos={0 0}, upload=485728]: Uploading piece... Dec 28 17:30:07.181298705 (554h1m7.05238175s) [upload=485728]: Next returned nil segment Dec 28 17:30:07.181320356 (554h1m7.052403401s) [upload=485728]: Waiting for error group managing segments... Dec 28 17:30:07.534065857 (554h1m7.405148902s) [noise=true , address=145.79.227.41:4085 , piece_id=436TDYTOAQQY2SISNNUQJPV7V4DWGFWMLSVIH7YVZ7FDVCI42THA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.539246893 (554h1m7.410329938s) [noise=true , address=145.79.227.37:4567 , piece_id=AN56FQIJVZHIEIIYOSQJEKPMDMXKLR4X6OBCOGKJK3DTHZ3LONBQ, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.539611677 (554h1m7.410694722s) [noise=true , address=145.79.227.4:4187 , piece_id=QI66LMP2CYC5R2DYJ5UPB7T4ALYGNZABZI2IEOWNWBOWSPXFIWVQ, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.540368468 (554h1m7.411451514s) [noise=true , address=145.79.227.36:4518 , piece_id=5BGPNQ2N7GWWYKGZZTWIGHNFL5TBQVI6OXWFBE5VGLENIXL2CM5Q, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.548398509 (554h1m7.419481554s) [noise=true , address=145.79.227.11:4729 , piece_id=PBEHUH44SSE2II72U27PVUQ3DUTWA3FWX35KCSDFBXD7LVB6CVWA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.551097618 (554h1m7.422180673s) [noise=true , address=145.79.227.3:4149 , piece_id=JGV2WF4WLQOMDTAEM6OPZZMLEXUSAVZYBXZGOMKRKOVFPBM66QSQ, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.553135048 (554h1m7.424218094s) [noise=true , address=145.79.227.2:4113 , piece_id=ED5WPPMQMSBKO5QXALSTXHKPZX3OYF6QDGRMOUUU2KZB5FCI35FA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.789722924 (554h1m7.660805969s) [noise=true , address=45.91.116.4:4939 , piece_id=LKH4Y3D7GFBBBUU7TFLHM7TUCK7477ZUC6JG2TJT5D32B5VG7O7Q, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.802140920 (554h1m7.673223966s) [noise=true , address=77.90.31.16:4736 , piece_id=PEJR4LHCWENG6LTUMUVBW5F5XMKSHRKGSRQWJ4TVBLKAUPKBXGPQ, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.814393281 (554h1m7.685476326s) [noise=true , address=77.90.31.11:4553 , piece_id=MMNX6C2HKQ3Z3TWQEET6DBSBEFME3AV7UQLDSGP6ZTCMXHKUEPBA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.834424064 (554h1m7.705507119s) [noise=true , address=45.91.116.11:4120 , piece_id=EOM5TZNJ4M4OAAOZ77F4IESINYF3JDRMN25LWS4TJJVCWQOJBJEA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.849053060 (554h1m7.720136105s) [noise=true , address=77.90.31.18:4808 , piece_id=35MESIN2JGX2LH6U4BKMH7XNIYKZODPKL6RRRSWTZYM2ZI5IQO5A, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.852486007 (554h1m7.723569052s) [noise=true , address=45.91.116.14:4190 , piece_id=XWQMNUGXN3VQYDTZOOTNHVQZ5AVIAW6TFWQTXJ4PU2GIYQMAPCYA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: <nil> Dec 28 17:30:07.852506165 (554h1m7.72358921s) [seg_pos={0 0}, upload=485728]: Segment reached optimal threshold of 13 pieces. Dec 28 17:30:07.853116497 (554h1m7.724199542s) [noise=true , address=45.91.116.5:4297 , piece_id=32ZEP2ET5EZXQWJ6TWQZBF3RZ2MOKJRYQAAPFNAVKANDBI5TFK5A, seg_pos={0 0}, upload=485728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 17:30:07.853127177 (554h1m7.724210223s) [noise=true , address=45.91.116.5:4297 , piece_id=32ZEP2ET5EZXQWJ6TWQZBF3RZ2MOKJRYQAAPFNAVKANDBI5TFK5A, seg_pos={0 0}, upload=485728]: Long tail cancellation due to optimal threshold Dec 28 17:30:07.853219854 (554h1m7.724302909s) [noise=true , address=45.91.116.7:4011 , piece_id=3RCTIMF3U7I65PEIF524MWYPARHOK4RVGBTMPHKBD43FLGAQRTCA, seg_pos={0 0}, upload=485728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: read tcp 188.166.225.170:38108->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:30:07.853231005 (554h1m7.72431407s) [noise=true , address=45.91.116.7:4011 , piece_id=3RCTIMF3U7I65PEIF524MWYPARHOK4RVGBTMPHKBD43FLGAQRTCA, seg_pos={0 0}, upload=485728]: Long tail cancellation due to optimal threshold Dec 28 17:30:07.853419664 (554h1m7.724502719s) [noise=true , address=77.90.31.14:4664 , piece_id=2C6FZTSSUGD5U7B2VUXLULA2ADXLFVLRC6GG5W5M7ZZWUW6PNZMQ, seg_pos={0 0}, upload=485728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:47712->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 17:30:07.853429413 (554h1m7.724512448s) [noise=true , address=77.90.31.14:4664 , piece_id=2C6FZTSSUGD5U7B2VUXLULA2ADXLFVLRC6GG5W5M7ZZWUW6PNZMQ, seg_pos={0 0}, upload=485728]: Long tail cancellation due to optimal threshold Dec 28 17:30:07.853485540 (554h1m7.724568575s) [seg_pos={0 0}, upload=485728]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:30:07.853597232 (554h1m7.724680277s) [upload=485728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:30:07.853604687 (554h1m7.724687722s) [upload=485728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:30:08.147634994 (554h1m8.018718039s) [upload=485728]: Done uploading segments... Dec 28 17:30:08.147666063 (554h1m8.018749098s) [upload=485728]: Upload finished. err: <nil> Duration: 30.003452656s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.....XXX.X.......X..X....................X..X............] duration: 3.470518375s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:30:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470548784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 43.311165601s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:29:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 43.311197522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX...] duration: 8.030694166s trace: https://jaeger.dev.storj.io/trace/6f1af8c90cf6b4f5 FAIL (2025-12-28 17:29:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ) context deadline exceeded ======LOGS====== Dec 28 17:25:50.170624061 (553h56m50.041707096s) [upload=485670]: Starting upload Dec 28 17:25:50.170691099 (553h56m50.041774154s) [upload=485670]: Done starting upload Dec 28 17:25:50.170717830 (553h56m50.041800865s) [upload=485670]: Uploading segments... Dec 28 17:25:50.170877353 (553h56m50.041960388s) [upload=485670]: Got next segment. Inline: false Dec 28 17:25:50.170906879 (553h56m50.041989914s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:25:50.170911228 (553h56m50.041994263s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:25:50.415502855 (553h56m50.28658589s) [seg_pos={0 0}, upload=485670]: Begin upload segment... Dec 28 17:25:50.415551298 (553h56m50.286634333s) [seg_pos={0 0}, upload=485670]: Done begin upload segment. Dec 28 17:25:50.415577938 (553h56m50.286660983s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415616711 (553h56m50.286699747s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415993098 (553h56m50.287076133s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416271869 (553h56m50.287354904s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416552283 (553h56m50.287635318s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416813261 (553h56m50.287896306s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417073425 (553h56m50.28815646s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417358238 (553h56m50.288441283s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417651807 (553h56m50.288734842s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418455117 (553h56m50.289538152s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418660388 (553h56m50.289743433s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419049999 (553h56m50.290133044s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419409093 (553h56m50.290492128s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419794637 (553h56m50.290877672s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.420432222 (553h56m50.291515267s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.423965599 (553h56m50.295048634s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:52.128376653 (553h56m51.999459698s) [upload=485670]: Next returned nil segment Dec 28 17:25:52.128388746 (553h56m51.999471781s) [upload=485670]: Waiting for error group managing segments... Dec 28 17:25:52.411556620 (553h56m52.282639665s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.419814203 (553h56m52.290897248s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424369538 (553h56m52.295452583s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424862767 (553h56m52.295945802s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.431593754 (553h56m52.302676799s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.434197622 (553h56m52.305280677s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.441314324 (553h56m52.312397369s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.443325133 (553h56m52.314408168s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.628231288 (553h56m52.499314343s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.670212558 (553h56m52.541295593s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.716773148 (553h56m52.587856193s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.723276902 (553h56m52.594359947s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.011976359 (553h56m52.883059404s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.012006206 (553h56m52.883089251s) [seg_pos={0 0}, upload=485670]: Segment reached optimal threshold of 13 pieces. Dec 28 17:25:53.012267333 (553h56m52.883350378s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012279817 (553h56m52.883362863s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012418041 (553h56m52.883501076s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012424152 (553h56m52.883507207s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012779178 (553h56m52.883862233s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012800098 (553h56m52.883883143s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012840014 (553h56m52.883923059s) [seg_pos={0 0}, upload=485670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:25:53.012968408 (553h56m52.884051443s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:25:53.012978348 (553h56m52.884061393s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:25:53.307382386 (553h56m53.178465431s) [upload=485670]: Done uploading segments... Dec 28 17:25:53.307430178 (553h56m53.178513213s) [upload=485670]: Upload finished. err: <nil> Duration: 8.030735675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.....XXX.X.......X..X....................X..X...........] duration: 4.65995231s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:29:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.65997879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X.....XXX.X.......X..X....................X..X..........] duration: 3.554123547s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:28:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554148064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 39.646420294s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 39.646453197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX..] duration: 7.677420664s trace: https://jaeger.dev.storj.io/trace/6f1af8c90cf6b4f5 FAIL (2025-12-28 17:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ) context deadline exceeded ======LOGS====== Dec 28 17:25:50.170624061 (553h56m50.041707096s) [upload=485670]: Starting upload Dec 28 17:25:50.170691099 (553h56m50.041774154s) [upload=485670]: Done starting upload Dec 28 17:25:50.170717830 (553h56m50.041800865s) [upload=485670]: Uploading segments... Dec 28 17:25:50.170877353 (553h56m50.041960388s) [upload=485670]: Got next segment. Inline: false Dec 28 17:25:50.170906879 (553h56m50.041989914s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:25:50.170911228 (553h56m50.041994263s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:25:50.415502855 (553h56m50.28658589s) [seg_pos={0 0}, upload=485670]: Begin upload segment... Dec 28 17:25:50.415551298 (553h56m50.286634333s) [seg_pos={0 0}, upload=485670]: Done begin upload segment. Dec 28 17:25:50.415577938 (553h56m50.286660983s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415616711 (553h56m50.286699747s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415993098 (553h56m50.287076133s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416271869 (553h56m50.287354904s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416552283 (553h56m50.287635318s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416813261 (553h56m50.287896306s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417073425 (553h56m50.28815646s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417358238 (553h56m50.288441283s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417651807 (553h56m50.288734842s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418455117 (553h56m50.289538152s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418660388 (553h56m50.289743433s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419049999 (553h56m50.290133044s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419409093 (553h56m50.290492128s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419794637 (553h56m50.290877672s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.420432222 (553h56m50.291515267s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.423965599 (553h56m50.295048634s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:52.128376653 (553h56m51.999459698s) [upload=485670]: Next returned nil segment Dec 28 17:25:52.128388746 (553h56m51.999471781s) [upload=485670]: Waiting for error group managing segments... Dec 28 17:25:52.411556620 (553h56m52.282639665s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.419814203 (553h56m52.290897248s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424369538 (553h56m52.295452583s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424862767 (553h56m52.295945802s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.431593754 (553h56m52.302676799s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.434197622 (553h56m52.305280677s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.441314324 (553h56m52.312397369s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.443325133 (553h56m52.314408168s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.628231288 (553h56m52.499314343s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.670212558 (553h56m52.541295593s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.716773148 (553h56m52.587856193s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.723276902 (553h56m52.594359947s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.011976359 (553h56m52.883059404s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.012006206 (553h56m52.883089251s) [seg_pos={0 0}, upload=485670]: Segment reached optimal threshold of 13 pieces. Dec 28 17:25:53.012267333 (553h56m52.883350378s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012279817 (553h56m52.883362863s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012418041 (553h56m52.883501076s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012424152 (553h56m52.883507207s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012779178 (553h56m52.883862233s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012800098 (553h56m52.883883143s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012840014 (553h56m52.883923059s) [seg_pos={0 0}, upload=485670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:25:53.012968408 (553h56m52.884051443s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:25:53.012978348 (553h56m52.884061393s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:25:53.307382386 (553h56m53.178465431s) [upload=485670]: Done uploading segments... Dec 28 17:25:53.307430178 (553h56m53.178513213s) [upload=485670]: Upload finished. err: <nil> Duration: 7.677450361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X....X.....XXX.X.......X..X....................X..X.........] duration: 3.468684404s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:27:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468713139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 15.487430324s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:27:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 15.487448999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........................X..................................] duration: 42.131842181s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 42.131870385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....X.....XXX.X.......X..X....................X..X........] duration: 3.412948504s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:27:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412978361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX.] duration: 6.205428411s trace: https://jaeger.dev.storj.io/trace/6f1af8c90cf6b4f5 FAIL (2025-12-28 17:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ) context deadline exceeded ======LOGS====== Dec 28 17:25:50.170624061 (553h56m50.041707096s) [upload=485670]: Starting upload Dec 28 17:25:50.170691099 (553h56m50.041774154s) [upload=485670]: Done starting upload Dec 28 17:25:50.170717830 (553h56m50.041800865s) [upload=485670]: Uploading segments... Dec 28 17:25:50.170877353 (553h56m50.041960388s) [upload=485670]: Got next segment. Inline: false Dec 28 17:25:50.170906879 (553h56m50.041989914s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:25:50.170911228 (553h56m50.041994263s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:25:50.415502855 (553h56m50.28658589s) [seg_pos={0 0}, upload=485670]: Begin upload segment... Dec 28 17:25:50.415551298 (553h56m50.286634333s) [seg_pos={0 0}, upload=485670]: Done begin upload segment. Dec 28 17:25:50.415577938 (553h56m50.286660983s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415616711 (553h56m50.286699747s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415993098 (553h56m50.287076133s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416271869 (553h56m50.287354904s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416552283 (553h56m50.287635318s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416813261 (553h56m50.287896306s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417073425 (553h56m50.28815646s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417358238 (553h56m50.288441283s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417651807 (553h56m50.288734842s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418455117 (553h56m50.289538152s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418660388 (553h56m50.289743433s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419049999 (553h56m50.290133044s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419409093 (553h56m50.290492128s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419794637 (553h56m50.290877672s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.420432222 (553h56m50.291515267s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.423965599 (553h56m50.295048634s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:52.128376653 (553h56m51.999459698s) [upload=485670]: Next returned nil segment Dec 28 17:25:52.128388746 (553h56m51.999471781s) [upload=485670]: Waiting for error group managing segments... Dec 28 17:25:52.411556620 (553h56m52.282639665s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.419814203 (553h56m52.290897248s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424369538 (553h56m52.295452583s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424862767 (553h56m52.295945802s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.431593754 (553h56m52.302676799s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.434197622 (553h56m52.305280677s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.441314324 (553h56m52.312397369s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.443325133 (553h56m52.314408168s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.628231288 (553h56m52.499314343s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.670212558 (553h56m52.541295593s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.716773148 (553h56m52.587856193s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.723276902 (553h56m52.594359947s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.011976359 (553h56m52.883059404s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.012006206 (553h56m52.883089251s) [seg_pos={0 0}, upload=485670]: Segment reached optimal threshold of 13 pieces. Dec 28 17:25:53.012267333 (553h56m52.883350378s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012279817 (553h56m52.883362863s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012418041 (553h56m52.883501076s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012424152 (553h56m52.883507207s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012779178 (553h56m52.883862233s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012800098 (553h56m52.883883143s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012840014 (553h56m52.883923059s) [seg_pos={0 0}, upload=485670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:25:53.012968408 (553h56m52.884051443s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:25:53.012978348 (553h56m52.884061393s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:25:53.307382386 (553h56m53.178465431s) [upload=485670]: Done uploading segments... Dec 28 17:25:53.307430178 (553h56m53.178513213s) [upload=485670]: Upload finished. err: <nil> Duration: 6.205449721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.X....X.....XXX.X.......X..X....................X..X.......] duration: 5.001701052s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:27:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00173162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........................X.................................] duration: 45.833120955s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 45.833151904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XXX] duration: 30.004452899s trace: https://jaeger.dev.storj.io/trace/6f1af8c90cf6b4f5 FAIL (2025-12-28 17:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ) context deadline exceeded ======LOGS====== Dec 28 17:25:50.170624061 (553h56m50.041707096s) [upload=485670]: Starting upload Dec 28 17:25:50.170691099 (553h56m50.041774154s) [upload=485670]: Done starting upload Dec 28 17:25:50.170717830 (553h56m50.041800865s) [upload=485670]: Uploading segments... Dec 28 17:25:50.170877353 (553h56m50.041960388s) [upload=485670]: Got next segment. Inline: false Dec 28 17:25:50.170906879 (553h56m50.041989914s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:25:50.170911228 (553h56m50.041994263s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:25:50.415502855 (553h56m50.28658589s) [seg_pos={0 0}, upload=485670]: Begin upload segment... Dec 28 17:25:50.415551298 (553h56m50.286634333s) [seg_pos={0 0}, upload=485670]: Done begin upload segment. Dec 28 17:25:50.415577938 (553h56m50.286660983s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415616711 (553h56m50.286699747s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.415993098 (553h56m50.287076133s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416271869 (553h56m50.287354904s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416552283 (553h56m50.287635318s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.416813261 (553h56m50.287896306s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417073425 (553h56m50.28815646s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417358238 (553h56m50.288441283s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.417651807 (553h56m50.288734842s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418455117 (553h56m50.289538152s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.418660388 (553h56m50.289743433s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419049999 (553h56m50.290133044s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419409093 (553h56m50.290492128s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.419794637 (553h56m50.290877672s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.420432222 (553h56m50.291515267s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:50.423965599 (553h56m50.295048634s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Uploading piece... Dec 28 17:25:52.128376653 (553h56m51.999459698s) [upload=485670]: Next returned nil segment Dec 28 17:25:52.128388746 (553h56m51.999471781s) [upload=485670]: Waiting for error group managing segments... Dec 28 17:25:52.411556620 (553h56m52.282639665s) [noise=true , address=145.79.227.16:4910 , piece_id=TT6TLVQ326OBK5YF5HTNR6CISIZRKV5OUZPFKA6O6B5UWCQNVKTQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.419814203 (553h56m52.290897248s) [noise=true , address=145.79.227.37:4567 , piece_id=WSG47OTRT665F3VE55OAFMNTVTIA5GJE3OOB2UZQ4E7RADUJ4TMA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424369538 (553h56m52.295452583s) [noise=true , address=145.79.227.19:4079 , piece_id=BFSOHTPW3ETN3CMCFBV7AHGLG4KX4YH63WOJTPI33QDGIC7YYYDQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.424862767 (553h56m52.295945802s) [noise=true , address=145.79.227.39:4014 , piece_id=LVFX2ZG5YBSLT3R77WDD2SD55AW7G6HZ2YFZ63IPB6FFQ3NDUUIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.431593754 (553h56m52.302676799s) [noise=true , address=145.79.227.51:4409 , piece_id=CC2UYM4LOZZ6HI5J6MFBWB5E7AFZANN36WN5XGG3LXGMDXAXSZIQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.434197622 (553h56m52.305280677s) [noise=true , address=145.79.227.15:4875 , piece_id=P43HORIBKGY7EPFGUZWAMQFSTK6CGL6SONP5IIOVW6O4EPNZSKOQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.441314324 (553h56m52.312397369s) [noise=true , address=145.79.227.47:4266 , piece_id=6PMNLHIAUBCZO7VYLE733N3UEIYRE2JUQENDS3ZL7U5DQ7QK7WRA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.443325133 (553h56m52.314408168s) [noise=true , address=145.79.227.3:4151 , piece_id=MTELYE6BHAPA4WOR3577SKRTBKKJ5J2UH44TRDHGW2CT5STGH7NA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.628231288 (553h56m52.499314343s) [noise=true , address=45.91.116.11:4120 , piece_id=NIQEGUHIAEHUZPU4VRW5GIB5O4WXK45J5OYGAE33QO32YXMNBPNA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.670212558 (553h56m52.541295593s) [noise=true , address=45.91.116.8:4047 , piece_id=VBND2ODH2TI7EOC3T2O7ZIOORUJZ63SVCTOWTRBQBFBGIOUKYROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.716773148 (553h56m52.587856193s) [noise=true , address=77.90.31.21:4878 , piece_id=DZCKV5UEDKI2JTAINIGI3PK73QSVR2IDOMQ724OAVLTGFJW5AU6A, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:52.723276902 (553h56m52.594359947s) [noise=true , address=77.90.31.12:4589 , piece_id=SI7CR6VFNRADGUDJXLK22XWLJB7DXJFTIILELSMLLFHUWUS3ZFJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.011976359 (553h56m52.883059404s) [noise=true , address=45.91.116.4:4937 , piece_id=QWSURSCYUIM7MD5CN234FG5E5V6RNT3HHUWPAQ62T744OYKSFROQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: <nil> Dec 28 17:25:53.012006206 (553h56m52.883089251s) [seg_pos={0 0}, upload=485670]: Segment reached optimal threshold of 13 pieces. Dec 28 17:25:53.012267333 (553h56m52.883350378s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012279817 (553h56m52.883362863s) [noise=true , address=45.91.116.7:4009 , piece_id=J6AWFB4W76OT4BAX77KQTE5J4ICHGFCFKN6CNO4CVARRNS6JYKJQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012418041 (553h56m52.883501076s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012424152 (553h56m52.883507207s) [noise=true , address=145.79.227.34:4906 , piece_id=TYGF4AKC6MLONKNIR5PGQZ2RQA5LTRHVXPHSKHPUYY2HFSHYDEXQ, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012779178 (553h56m52.883862233s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:53.012800098 (553h56m52.883883143s) [noise=true , address=45.91.116.12:4228 , piece_id=XAQBYBMZHHQU66BN6PEZQJFF5KSHN5DAIKZWUG5QXLDWQVGKLHKA, seg_pos={0 0}, upload=485670]: Long tail cancellation due to optimal threshold Dec 28 17:25:53.012840014 (553h56m52.883923059s) [seg_pos={0 0}, upload=485670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:25:53.012968408 (553h56m52.884051443s) [upload=485670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:25:53.012978348 (553h56m52.884061393s) [upload=485670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:25:53.307382386 (553h56m53.178465431s) [upload=485670]: Done uploading segments... Dec 28 17:25:53.307430178 (553h56m53.178513213s) [upload=485670]: Upload finished. err: <nil> Duration: 30.004465884s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.X....X.....XXX.X.......X..X....................X..X......] duration: 3.55164613s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:26:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551673813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.X.X....X.....XXX.X.......X..X....................X..X.....] duration: 3.531103022s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:26:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531126295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.XX] duration: 30.002147227s trace: https://jaeger.dev.storj.io/trace/697a4dcc603857e7 FAIL (2025-12-28 17:25:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:25:19.954178711 (553h56m19.825261756s) [upload=485666]: Starting upload Dec 28 17:25:19.954251029 (553h56m19.825334064s) [upload=485666]: Done starting upload Dec 28 17:25:19.954380706 (553h56m19.825463741s) [upload=485666]: Uploading segments... Dec 28 17:25:19.954452483 (553h56m19.825535528s) [upload=485666]: Got next segment. Inline: false Dec 28 17:25:19.954486598 (553h56m19.825569643s) [upload=485666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:25:19.954492158 (553h56m19.825575203s) [upload=485666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:25:20.225474018 (553h56m20.096557053s) [seg_pos={0 0}, upload=485666]: Begin upload segment... Dec 28 17:25:20.225532309 (553h56m20.096615344s) [seg_pos={0 0}, upload=485666]: Done begin upload segment. Dec 28 17:25:20.225569440 (553h56m20.096652475s) [noise=true , address=45.91.116.8:4045 , piece_id=3RLPJPMOP7AAQUSZAWI6P777DSFNLWHYJFWL6YAJ32TYBYR2BZUQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.225581603 (553h56m20.096664638s) [noise=true , address=145.79.227.19:4077 , piece_id=4PS6Q6XU4NMOZWH5I3S6PWSWE2CNJRDB6F5I42NA5FVOKIYRKYEQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.225694198 (553h56m20.096777233s) [noise=true , address=145.79.227.36:4517 , piece_id=HXJU244YQVXYUCZJ3DSYIZIQIYJX5UJDV24WSVRTVK47J3SWHDNQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.226687779 (553h56m20.097770824s) [noise=true , address=45.91.116.9:4262 , piece_id=XH5R4L4XW4XGEGLQRIQ67SJMQJTFEZO4EJIVGH3XEA6XWUFFTCSA, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.227268716 (553h56m20.098351751s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGIVYW7SNMCB47HV5DMIWNMPBF3XEG4PPKATOQA5HKB4SVOHRUUQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.227665741 (553h56m20.098748786s) [noise=true , address=145.79.227.50:4375 , piece_id=RGURAAHFJ4QXRORQVH2OVMJ5GJZXEBZZB3C5OOGOPMWR47PIXAQQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.227995740 (553h56m20.099078785s) [noise=true , address=77.90.31.18:4805 , piece_id=YA527MHPFG4WICA3T6QZQONVXYONJMKZ3SCJKSVUFNPYTPV2BXLA, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.228069270 (553h56m20.099152305s) [noise=true , address=145.79.227.21:4473 , piece_id=M573H4GQO44QF6MKHC2DNCWSKOM37HOQ6UK6EDTYEIO7MNJD57XQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.228317893 (553h56m20.099400928s) [noise=true , address=45.91.116.5:4299 , piece_id=BGV2TKOBJPBX2MA27ABBRBNMKTS6RYEBESI26367N4RFTRWPPAXQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.228635979 (553h56m20.099719024s) [noise=true , address=145.79.227.52:4446 , piece_id=SYK7KYRKD4AM43AQ5MSTI4J2DFLGCLVP6EEHZ3NWML3Z5DDUPWNA, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.228747210 (553h56m20.099830256s) [noise=true , address=145.79.227.45:4231 , piece_id=WV53RK5DZULLZRUDDMHDGMZU6FMLI2L6N3EF7TNUC7DYC5Z5MA4A, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.228943475 (553h56m20.10002651s) [noise=true , address=145.79.227.5:4221 , piece_id=G2ID2FHH35UI5TUS6SFVM2LPB3RLSSM36PTCDX3UBQW3Y5J7K6JQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.229235681 (553h56m20.100318726s) [noise=true , address=77.90.31.14:4663 , piece_id=36HQNWXKK544UKT7LPXOTDMZMYXWEXJIBQOIKK3E5IO5USP6PGDA, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.229260228 (553h56m20.100343263s) [noise=true , address=145.79.227.17:4972 , piece_id=22PQWTAUKMUEVQ2F2N2BHHEN3H3XX6X5XEM7Q6C3MHB5XBOQNKNQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.229581519 (553h56m20.100664564s) [noise=true , address=77.90.31.15:4697 , piece_id=FMNVVDNTGVNQ23CO7O57ZYRKIX54GQ6EQYEYW4AUN3UDMUC5KOSQ, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:20.230005216 (553h56m20.101088251s) [noise=true , address=145.79.227.51:4410 , piece_id=ZONZBHP2GZZ4M57UL2LGEE6SHFLKIYNPSKYHRSNCE3QDQUAC3GIA, seg_pos={0 0}, upload=485666]: Uploading piece... Dec 28 17:25:21.889434334 (553h56m21.760517369s) [upload=485666]: Next returned nil segment Dec 28 17:25:21.889448741 (553h56m21.760531786s) [upload=485666]: Waiting for error group managing segments... Dec 28 17:25:22.226347619 (553h56m22.097430664s) [noise=true , address=145.79.227.36:4517 , piece_id=HXJU244YQVXYUCZJ3DSYIZIQIYJX5UJDV24WSVRTVK47J3SWHDNQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.231468731 (553h56m22.102551776s) [noise=true , address=145.79.227.52:4446 , piece_id=SYK7KYRKD4AM43AQ5MSTI4J2DFLGCLVP6EEHZ3NWML3Z5DDUPWNA, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.232024459 (553h56m22.103107494s) [noise=true , address=145.79.227.45:4231 , piece_id=WV53RK5DZULLZRUDDMHDGMZU6FMLI2L6N3EF7TNUC7DYC5Z5MA4A, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.238981547 (553h56m22.110064592s) [noise=true , address=145.79.227.5:4221 , piece_id=G2ID2FHH35UI5TUS6SFVM2LPB3RLSSM36PTCDX3UBQW3Y5J7K6JQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.247514665 (553h56m22.118597711s) [noise=true , address=145.79.227.21:4473 , piece_id=M573H4GQO44QF6MKHC2DNCWSKOM37HOQ6UK6EDTYEIO7MNJD57XQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.253419900 (553h56m22.124502935s) [noise=true , address=145.79.227.51:4410 , piece_id=ZONZBHP2GZZ4M57UL2LGEE6SHFLKIYNPSKYHRSNCE3QDQUAC3GIA, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.261177693 (553h56m22.132260738s) [noise=true , address=145.79.227.42:4123 , piece_id=ZGIVYW7SNMCB47HV5DMIWNMPBF3XEG4PPKATOQA5HKB4SVOHRUUQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.267297816 (553h56m22.138380851s) [noise=true , address=145.79.227.17:4972 , piece_id=22PQWTAUKMUEVQ2F2N2BHHEN3H3XX6X5XEM7Q6C3MHB5XBOQNKNQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.271905610 (553h56m22.142988655s) [noise=true , address=145.79.227.50:4375 , piece_id=RGURAAHFJ4QXRORQVH2OVMJ5GJZXEBZZB3C5OOGOPMWR47PIXAQQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.429948213 (553h56m22.301031258s) [noise=true , address=77.90.31.18:4805 , piece_id=YA527MHPFG4WICA3T6QZQONVXYONJMKZ3SCJKSVUFNPYTPV2BXLA, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.468007104 (553h56m22.339090139s) [noise=true , address=45.91.116.5:4299 , piece_id=BGV2TKOBJPBX2MA27ABBRBNMKTS6RYEBESI26367N4RFTRWPPAXQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.474127878 (553h56m22.345210923s) [noise=true , address=45.91.116.8:4045 , piece_id=3RLPJPMOP7AAQUSZAWI6P777DSFNLWHYJFWL6YAJ32TYBYR2BZUQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.501946060 (553h56m22.373029095s) [noise=true , address=77.90.31.14:4663 , piece_id=36HQNWXKK544UKT7LPXOTDMZMYXWEXJIBQOIKK3E5IO5USP6PGDA, seg_pos={0 0}, upload=485666]: Done uploading piece. err: <nil> Dec 28 17:25:22.501971819 (553h56m22.373054854s) [seg_pos={0 0}, upload=485666]: Segment reached optimal threshold of 13 pieces. Dec 28 17:25:22.502187119 (553h56m22.373270154s) [noise=true , address=45.91.116.9:4262 , piece_id=XH5R4L4XW4XGEGLQRIQ67SJMQJTFEZO4EJIVGH3XEA6XWUFFTCSA, seg_pos={0 0}, upload=485666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:22.502197679 (553h56m22.373280714s) [noise=true , address=45.91.116.9:4262 , piece_id=XH5R4L4XW4XGEGLQRIQ67SJMQJTFEZO4EJIVGH3XEA6XWUFFTCSA, seg_pos={0 0}, upload=485666]: Long tail cancellation due to optimal threshold Dec 28 17:25:22.502262342 (553h56m22.373345387s) [noise=true , address=77.90.31.15:4697 , piece_id=FMNVVDNTGVNQ23CO7O57ZYRKIX54GQ6EQYEYW4AUN3UDMUC5KOSQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:34918->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:34918->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34918->77.90.31.15:4697: use of closed network connection Dec 28 17:25:22.502273333 (553h56m22.373356378s) [noise=true , address=77.90.31.15:4697 , piece_id=FMNVVDNTGVNQ23CO7O57ZYRKIX54GQ6EQYEYW4AUN3UDMUC5KOSQ, seg_pos={0 0}, upload=485666]: Long tail cancellation due to optimal threshold Dec 28 17:25:22.502436133 (553h56m22.373519178s) [noise=true , address=145.79.227.19:4077 , piece_id=4PS6Q6XU4NMOZWH5I3S6PWSWE2CNJRDB6F5I42NA5FVOKIYRKYEQ, seg_pos={0 0}, upload=485666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 28 17:25:22.502443106 (553h56m22.373526141s) [noise=true , address=145.79.227.19:4077 , piece_id=4PS6Q6XU4NMOZWH5I3S6PWSWE2CNJRDB6F5I42NA5FVOKIYRKYEQ, seg_pos={0 0}, upload=485666]: Long tail cancellation due to optimal threshold Dec 28 17:25:22.502536133 (553h56m22.373619179s) [seg_pos={0 0}, upload=485666]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:25:22.502619482 (553h56m22.373702527s) [upload=485666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:25:22.502625603 (553h56m22.373708658s) [upload=485666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:25:22.836766611 (553h56m22.707849656s) [upload=485666]: Done uploading segments... Dec 28 17:25:22.836790075 (553h56m22.70787312s) [upload=485666]: Upload finished. err: <nil> Duration: 30.00217005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........................X................................] duration: 34.351109142s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 34.351136966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XX.X.X....X.....XXX.X.......X..X....................X..X....] duration: 3.4987041s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:25:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498871086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........................X...............................] duration: 50.038930672s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 50.038964857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXX.X.X....X.....XXX.X.......X..X....................X..X...] duration: 3.48450485s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:24:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484527874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.X] duration: 30.002772692s trace: https://jaeger.dev.storj.io/trace/66e21e10bbed18d7 FAIL (2025-12-28 17:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 22RBEZ7LVR6V3VMKA3DJYI2BZQV6MR4E34ZZ6NCX3R5AB2ZNQCCQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S35MRJ57DIOTIPWTFLSCCLINA4QVK4UZQQA7P47466GH5VQQXYRQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: KT6WDNOZT3IPMYEJVKVDBRJA2IRR7RBTKM4VKUEOALQRSITWV24Q) context deadline exceeded ======LOGS====== Dec 28 17:24:11.083912184 (553h55m10.954995219s) [upload=485651]: Starting upload Dec 28 17:24:11.084111684 (553h55m10.955194749s) [upload=485651]: Done starting upload Dec 28 17:24:11.084235149 (553h55m10.955318194s) [upload=485651]: Uploading segments... Dec 28 17:24:11.084439327 (553h55m10.955522373s) [upload=485651]: Got next segment. Inline: false Dec 28 17:24:11.084487410 (553h55m10.955570445s) [upload=485651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:24:11.084494613 (553h55m10.955577658s) [upload=485651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:24:11.333697402 (553h55m11.204780467s) [seg_pos={0 0}, upload=485651]: Begin upload segment... Dec 28 17:24:11.333752596 (553h55m11.204835641s) [seg_pos={0 0}, upload=485651]: Done begin upload segment. Dec 28 17:24:11.333786801 (553h55m11.204869846s) [noise=true , address=77.90.31.17:4770 , piece_id=USPRTHTGKW7TWQVS5AJMV6YDF5VUVY5GYRR4DYNRX3ETGWGBXYXQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.334185491 (553h55m11.205268536s) [noise=true , address=45.91.116.9:4262 , piece_id=S35MRJ57DIOTIPWTFLSCCLINA4QVK4UZQQA7P47466GH5VQQXYRQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.334700782 (553h55m11.205783827s) [noise=true , address=145.79.227.15:4875 , piece_id=TDES3R2HT4ZZRIWEPTQUHDCBD2Q2KRLFHU343EVLJB6GHXR7CJXA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.335300354 (553h55m11.206383399s) [noise=true , address=45.91.116.5:4297 , piece_id=F37RZKCXKX7XPMTSA3XM4SZH5CGBSU5ESIKATIP3JMQATJH3KPUA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.335941214 (553h55m11.207024259s) [noise=true , address=145.79.227.14:4837 , piece_id=XVV6GOEG3UEP56WE7VYY5FH6W7G6S6XB34XH2LOBTBCQ5PYVL7CQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.336821390 (553h55m11.207904435s) [noise=true , address=145.79.227.7:4295 , piece_id=XB5PT5TPODK7J2LVEAIJWUNN5SUELW7DAMPC4LSEHOEZEVPGGVKA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.336903356 (553h55m11.207986401s) [noise=true , address=145.79.227.52:4445 , piece_id=QIT2AZJS6QN7LXTCVCQCPU3WQT64FLXLPZINITLMUQD7ZLCKBXIQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.337230649 (553h55m11.208313694s) [noise=true , address=145.79.227.16:4910 , piece_id=FRXLXMNQRJAQUY45CHQ77Z6MUIR4BEMIRZNNIMZGDZ7FHT3U3VMA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.337415281 (553h55m11.208498326s) [noise=true , address=145.79.227.36:4519 , piece_id=PDI3EQNR2L3GI6TNA5PV3L66TDKNMBZZP7TKLB6LAH764I5KAWNA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.337602157 (553h55m11.208685202s) [noise=true , address=145.79.227.19:4078 , piece_id=Y6YYSOWA72VD5BMXGNJE6QVHS373IJ7WAP2XWPD2IE2TU4533JEQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.338062212 (553h55m11.209145258s) [noise=true , address=145.79.227.38:4977 , piece_id=L6ZBMPJ4QFRJ6OUTGB6Z365EB6KZNCRHTLCAHMWJ7TIU2IBBOQRA, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.338252606 (553h55m11.209335651s) [noise=true , address=45.91.116.10:4084 , piece_id=KT6WDNOZT3IPMYEJVKVDBRJA2IRR7RBTKM4VKUEOALQRSITWV24Q, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.338539732 (553h55m11.209622777s) [noise=true , address=45.91.116.11:4118 , piece_id=IRLOYK7SRSOE4ZAE4BPAQGDS6ICOYOHTNFSF3MXCL4CRDCJFBPGQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.338573536 (553h55m11.209656582s) [noise=true , address=145.79.227.6:4258 , piece_id=KINWWNGZEYU67UKHE6OCFX77YXNPIRAAGCYRN2Q7W37AUTJECALQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.338819966 (553h55m11.209903011s) [noise=true , address=77.90.31.15:4697 , piece_id=XPP3TGGKH72JUPZTF46STHW6VHMPFPJYUVSKAINQ4JRZVZWEDE7A, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:11.339078077 (553h55m11.210161122s) [noise=true , address=45.91.116.13:4156 , piece_id=22RBEZ7LVR6V3VMKA3DJYI2BZQV6MR4E34ZZ6NCX3R5AB2ZNQCCQ, seg_pos={0 0}, upload=485651]: Uploading piece... Dec 28 17:24:12.948221201 (553h55m12.819304247s) [upload=485651]: Next returned nil segment Dec 28 17:24:12.948234536 (553h55m12.819317572s) [upload=485651]: Waiting for error group managing segments... Dec 28 17:24:13.335739066 (553h55m13.206822101s) [noise=true , address=145.79.227.19:4078 , piece_id=Y6YYSOWA72VD5BMXGNJE6QVHS373IJ7WAP2XWPD2IE2TU4533JEQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.336289084 (553h55m13.207372119s) [noise=true , address=145.79.227.38:4977 , piece_id=L6ZBMPJ4QFRJ6OUTGB6Z365EB6KZNCRHTLCAHMWJ7TIU2IBBOQRA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.344631860 (553h55m13.215714895s) [noise=true , address=145.79.227.15:4875 , piece_id=TDES3R2HT4ZZRIWEPTQUHDCBD2Q2KRLFHU343EVLJB6GHXR7CJXA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.345404783 (553h55m13.216487828s) [noise=true , address=145.79.227.6:4258 , piece_id=KINWWNGZEYU67UKHE6OCFX77YXNPIRAAGCYRN2Q7W37AUTJECALQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.367076429 (553h55m13.238159474s) [noise=true , address=145.79.227.7:4295 , piece_id=XB5PT5TPODK7J2LVEAIJWUNN5SUELW7DAMPC4LSEHOEZEVPGGVKA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.367201918 (553h55m13.238284953s) [noise=true , address=145.79.227.36:4519 , piece_id=PDI3EQNR2L3GI6TNA5PV3L66TDKNMBZZP7TKLB6LAH764I5KAWNA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.372131584 (553h55m13.24321463s) [noise=true , address=145.79.227.52:4445 , piece_id=QIT2AZJS6QN7LXTCVCQCPU3WQT64FLXLPZINITLMUQD7ZLCKBXIQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.387175021 (553h55m13.258258076s) [noise=true , address=145.79.227.16:4910 , piece_id=FRXLXMNQRJAQUY45CHQ77Z6MUIR4BEMIRZNNIMZGDZ7FHT3U3VMA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.399032500 (553h55m13.270115545s) [noise=true , address=145.79.227.14:4837 , piece_id=XVV6GOEG3UEP56WE7VYY5FH6W7G6S6XB34XH2LOBTBCQ5PYVL7CQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.584746775 (553h55m13.45582981s) [noise=true , address=77.90.31.15:4697 , piece_id=XPP3TGGKH72JUPZTF46STHW6VHMPFPJYUVSKAINQ4JRZVZWEDE7A, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.598371448 (553h55m13.469454483s) [noise=true , address=45.91.116.10:4084 , piece_id=KT6WDNOZT3IPMYEJVKVDBRJA2IRR7RBTKM4VKUEOALQRSITWV24Q, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.606219563 (553h55m13.477302608s) [noise=true , address=77.90.31.17:4770 , piece_id=USPRTHTGKW7TWQVS5AJMV6YDF5VUVY5GYRR4DYNRX3ETGWGBXYXQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.616715949 (553h55m13.487798994s) [noise=true , address=45.91.116.9:4262 , piece_id=S35MRJ57DIOTIPWTFLSCCLINA4QVK4UZQQA7P47466GH5VQQXYRQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.616736248 (553h55m13.487819283s) [seg_pos={0 0}, upload=485651]: Segment reached optimal threshold of 13 pieces. Dec 28 17:24:13.616953522 (553h55m13.488036567s) [noise=true , address=45.91.116.5:4297 , piece_id=F37RZKCXKX7XPMTSA3XM4SZH5CGBSU5ESIKATIP3JMQATJH3KPUA, seg_pos={0 0}, upload=485651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 17:24:13.616963080 (553h55m13.488046125s) [noise=true , address=45.91.116.5:4297 , piece_id=F37RZKCXKX7XPMTSA3XM4SZH5CGBSU5ESIKATIP3JMQATJH3KPUA, seg_pos={0 0}, upload=485651]: Long tail cancellation due to optimal threshold Dec 28 17:24:13.617106783 (553h55m13.488189818s) [noise=true , address=45.91.116.13:4156 , piece_id=22RBEZ7LVR6V3VMKA3DJYI2BZQV6MR4E34ZZ6NCX3R5AB2ZNQCCQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: <nil> Dec 28 17:24:13.617317404 (553h55m13.488400459s) [noise=true , address=45.91.116.11:4118 , piece_id=IRLOYK7SRSOE4ZAE4BPAQGDS6ICOYOHTNFSF3MXCL4CRDCJFBPGQ, seg_pos={0 0}, upload=485651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 17:24:13.617326152 (553h55m13.488409187s) [noise=true , address=45.91.116.11:4118 , piece_id=IRLOYK7SRSOE4ZAE4BPAQGDS6ICOYOHTNFSF3MXCL4CRDCJFBPGQ, seg_pos={0 0}, upload=485651]: Long tail cancellation due to optimal threshold Dec 28 17:24:13.617343534 (553h55m13.488426569s) [seg_pos={0 0}, upload=485651]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 17:24:13.617425321 (553h55m13.488508356s) [upload=485651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:24:13.617430410 (553h55m13.488513465s) [upload=485651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:24:13.916247384 (553h55m13.787330439s) [upload=485651]: Done uploading segments... Dec 28 17:24:13.916272893 (553h55m13.787355938s) [upload=485651]: Upload finished. err: <nil> Duration: 30.002795585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X........................X..............................] duration: 48.368406092s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 48.368434105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXX.X.X....X.....XXX.X.......X..X....................X..X..] duration: 3.497733067s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:23:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497762833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX.] duration: 7.838772217s trace: https://jaeger.dev.storj.io/trace/7bcd27f7d3027394 FAIL (2025-12-28 17:23:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:22:48.981301375 (553h53m48.852384411s) [upload=485631]: Starting upload Dec 28 17:22:48.981370136 (553h53m48.852453182s) [upload=485631]: Done starting upload Dec 28 17:22:48.981429960 (553h53m48.852513005s) [upload=485631]: Uploading segments... Dec 28 17:22:48.981572691 (553h53m48.852655736s) [upload=485631]: Got next segment. Inline: false Dec 28 17:22:48.981605324 (553h53m48.852688369s) [upload=485631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:22:48.981609672 (553h53m48.852692717s) [upload=485631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:22:49.288594685 (553h53m49.159677731s) [seg_pos={0 0}, upload=485631]: Begin upload segment... Dec 28 17:22:49.288647415 (553h53m49.15973046s) [seg_pos={0 0}, upload=485631]: Done begin upload segment. Dec 28 17:22:49.288681851 (553h53m49.159764896s) [noise=true , address=77.90.31.21:4878 , piece_id=TUJMMRDRXLQXLPU3BZZGTWNDM3WAH6JN2BYWRUAR7FUJ2MY3SSVA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.288799385 (553h53m49.15988243s) [noise=true , address=45.91.116.13:4154 , piece_id=URRSA6XX7YCXYDECIMSQ7I7FX2VFHMZBXKL3KHWQSSQP2OOH5LWQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.289352177 (553h53m49.160435212s) [noise=true , address=145.79.227.6:4258 , piece_id=YVJZARZ7B5BZYZYXEOTUCE2RQZVZRMYLZBPJR4IQGQUPSTUCOIMA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.289772869 (553h53m49.160855904s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.290371849 (553h53m49.161454884s) [noise=true , address=145.79.227.14:4840 , piece_id=V5SMBB2BUBS3LKFJBJNSZJEUMMTAWJ5OQ7Y443THDR5AO6VAQ3UA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.290774786 (553h53m49.161857821s) [noise=true , address=145.79.227.11:4732 , piece_id=6MMYMMDHMKVAGT3WUOABDPFAFOSW4MKVP6QRXCHR5HX5NIB5FOMQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.291177513 (553h53m49.162260548s) [noise=true , address=45.91.116.5:4297 , piece_id=5NXKOWBUM74XCJI5Q7SANUOZDRRE7L662FOIS5QK7RIDDLZXFCMA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.291617221 (553h53m49.162700256s) [noise=true , address=145.79.227.16:4910 , piece_id=BGLPWLKNLILD42VIFR6PHQAVWWEIOOAZI6EEMH3264ZUNFG2GGUQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.292316543 (553h53m49.163399598s) [noise=true , address=145.79.227.3:4149 , piece_id=JPMIXVLL4L3ABJNN7TXMCKOGDRJVEQTL3G2FHJ3DVQB57OZEILOA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.293936447 (553h53m49.165019482s) [noise=true , address=145.79.227.39:4013 , piece_id=6622XCEZTSBLNR24JQ2T5VANM4PREH5YASZLHPRWQPOYDEAEPSYQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.293948549 (553h53m49.165031584s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.295498351 (553h53m49.166581396s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.296887747 (553h53m49.167970782s) [noise=true , address=145.79.227.50:4375 , piece_id=QDHDKKMNUJNIPISLFTNAGYOU4JZ34NWCQGAL2FABLTIYOSWYS57Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.298476242 (553h53m49.169559277s) [noise=true , address=77.90.31.19:4842 , piece_id=Z4OCWJB4VIAPWCIBKPGQABK7K5XIDSU5IDRRCI5KXDGFRTNKJ7JQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.300312888 (553h53m49.171395923s) [noise=true , address=145.79.227.38:4977 , piece_id=QV2YZMQRL5ZR227NBTWCXF5DJSHHJ62MCCACZIQLYA7WB4VQ6BSQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.301792276 (553h53m49.172875321s) [noise=true , address=145.79.227.47:4265 , piece_id=Y772Y5ZFFDCHRRE5WAB4XNNL2TQ7WQCSGLBFBXHH5SSNG65JXJ2Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:50.913044666 (553h53m50.784127711s) [upload=485631]: Next returned nil segment Dec 28 17:22:50.913056710 (553h53m50.784139745s) [upload=485631]: Waiting for error group managing segments... Dec 28 17:22:51.295865198 (553h53m51.166948243s) [noise=true , address=145.79.227.6:4258 , piece_id=YVJZARZ7B5BZYZYXEOTUCE2RQZVZRMYLZBPJR4IQGQUPSTUCOIMA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.296068094 (553h53m51.167151139s) [noise=true , address=145.79.227.11:4732 , piece_id=6MMYMMDHMKVAGT3WUOABDPFAFOSW4MKVP6QRXCHR5HX5NIB5FOMQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.303940945 (553h53m51.17502398s) [noise=true , address=145.79.227.38:4977 , piece_id=QV2YZMQRL5ZR227NBTWCXF5DJSHHJ62MCCACZIQLYA7WB4VQ6BSQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.313392253 (553h53m51.184475298s) [noise=true , address=145.79.227.50:4375 , piece_id=QDHDKKMNUJNIPISLFTNAGYOU4JZ34NWCQGAL2FABLTIYOSWYS57Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.317417217 (553h53m51.188500262s) [noise=true , address=145.79.227.3:4149 , piece_id=JPMIXVLL4L3ABJNN7TXMCKOGDRJVEQTL3G2FHJ3DVQB57OZEILOA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.329102457 (553h53m51.200185502s) [noise=true , address=145.79.227.39:4013 , piece_id=6622XCEZTSBLNR24JQ2T5VANM4PREH5YASZLHPRWQPOYDEAEPSYQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.336660270 (553h53m51.207743305s) [noise=true , address=145.79.227.14:4840 , piece_id=V5SMBB2BUBS3LKFJBJNSZJEUMMTAWJ5OQ7Y443THDR5AO6VAQ3UA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.338750239 (553h53m51.209833284s) [noise=true , address=145.79.227.47:4265 , piece_id=Y772Y5ZFFDCHRRE5WAB4XNNL2TQ7WQCSGLBFBXHH5SSNG65JXJ2Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.355191857 (553h53m51.226274902s) [noise=true , address=145.79.227.16:4910 , piece_id=BGLPWLKNLILD42VIFR6PHQAVWWEIOOAZI6EEMH3264ZUNFG2GGUQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.562580693 (553h53m51.433663728s) [noise=true , address=77.90.31.21:4878 , piece_id=TUJMMRDRXLQXLPU3BZZGTWNDM3WAH6JN2BYWRUAR7FUJ2MY3SSVA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.866792589 (553h53m51.737875624s) [noise=true , address=45.91.116.13:4154 , piece_id=URRSA6XX7YCXYDECIMSQ7I7FX2VFHMZBXKL3KHWQSSQP2OOH5LWQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.147275309 (553h53m52.018358344s) [noise=true , address=77.90.31.19:4842 , piece_id=Z4OCWJB4VIAPWCIBKPGQABK7K5XIDSU5IDRRCI5KXDGFRTNKJ7JQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.330287198 (553h53m52.201370253s) [noise=true , address=45.91.116.5:4297 , piece_id=5NXKOWBUM74XCJI5Q7SANUOZDRRE7L662FOIS5QK7RIDDLZXFCMA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.330307748 (553h53m52.201390783s) [seg_pos={0 0}, upload=485631]: Segment reached optimal threshold of 13 pieces. Dec 28 17:22:52.330496747 (553h53m52.201579802s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.330506065 (553h53m52.20158911s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332295342 (553h53m52.203378387s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.332311232 (553h53m52.203394267s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332463042 (553h53m52.203546097s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.332471287 (553h53m52.203554332s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332489581 (553h53m52.203572627s) [seg_pos={0 0}, upload=485631]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:22:52.332568272 (553h53m52.203651317s) [upload=485631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:22:52.332574925 (553h53m52.20365796s) [upload=485631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:22:52.615749100 (553h53m52.486832135s) [upload=485631]: Done uploading segments... Dec 28 17:22:52.615766212 (553h53m52.486849248s) [upload=485631]: Upload finished. err: <nil> Duration: 7.838793937s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XXXX.X.X....X.....XXX.X.......X..X....................X..X.] duration: 3.391687579s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:23:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.391720752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XXX] duration: 30.004499472s trace: https://jaeger.dev.storj.io/trace/7bcd27f7d3027394 FAIL (2025-12-28 17:23:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 17:22:48.981301375 (553h53m48.852384411s) [upload=485631]: Starting upload Dec 28 17:22:48.981370136 (553h53m48.852453182s) [upload=485631]: Done starting upload Dec 28 17:22:48.981429960 (553h53m48.852513005s) [upload=485631]: Uploading segments... Dec 28 17:22:48.981572691 (553h53m48.852655736s) [upload=485631]: Got next segment. Inline: false Dec 28 17:22:48.981605324 (553h53m48.852688369s) [upload=485631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:22:48.981609672 (553h53m48.852692717s) [upload=485631]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:22:49.288594685 (553h53m49.159677731s) [seg_pos={0 0}, upload=485631]: Begin upload segment... Dec 28 17:22:49.288647415 (553h53m49.15973046s) [seg_pos={0 0}, upload=485631]: Done begin upload segment. Dec 28 17:22:49.288681851 (553h53m49.159764896s) [noise=true , address=77.90.31.21:4878 , piece_id=TUJMMRDRXLQXLPU3BZZGTWNDM3WAH6JN2BYWRUAR7FUJ2MY3SSVA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.288799385 (553h53m49.15988243s) [noise=true , address=45.91.116.13:4154 , piece_id=URRSA6XX7YCXYDECIMSQ7I7FX2VFHMZBXKL3KHWQSSQP2OOH5LWQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.289352177 (553h53m49.160435212s) [noise=true , address=145.79.227.6:4258 , piece_id=YVJZARZ7B5BZYZYXEOTUCE2RQZVZRMYLZBPJR4IQGQUPSTUCOIMA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.289772869 (553h53m49.160855904s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.290371849 (553h53m49.161454884s) [noise=true , address=145.79.227.14:4840 , piece_id=V5SMBB2BUBS3LKFJBJNSZJEUMMTAWJ5OQ7Y443THDR5AO6VAQ3UA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.290774786 (553h53m49.161857821s) [noise=true , address=145.79.227.11:4732 , piece_id=6MMYMMDHMKVAGT3WUOABDPFAFOSW4MKVP6QRXCHR5HX5NIB5FOMQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.291177513 (553h53m49.162260548s) [noise=true , address=45.91.116.5:4297 , piece_id=5NXKOWBUM74XCJI5Q7SANUOZDRRE7L662FOIS5QK7RIDDLZXFCMA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.291617221 (553h53m49.162700256s) [noise=true , address=145.79.227.16:4910 , piece_id=BGLPWLKNLILD42VIFR6PHQAVWWEIOOAZI6EEMH3264ZUNFG2GGUQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.292316543 (553h53m49.163399598s) [noise=true , address=145.79.227.3:4149 , piece_id=JPMIXVLL4L3ABJNN7TXMCKOGDRJVEQTL3G2FHJ3DVQB57OZEILOA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.293936447 (553h53m49.165019482s) [noise=true , address=145.79.227.39:4013 , piece_id=6622XCEZTSBLNR24JQ2T5VANM4PREH5YASZLHPRWQPOYDEAEPSYQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.293948549 (553h53m49.165031584s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.295498351 (553h53m49.166581396s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.296887747 (553h53m49.167970782s) [noise=true , address=145.79.227.50:4375 , piece_id=QDHDKKMNUJNIPISLFTNAGYOU4JZ34NWCQGAL2FABLTIYOSWYS57Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.298476242 (553h53m49.169559277s) [noise=true , address=77.90.31.19:4842 , piece_id=Z4OCWJB4VIAPWCIBKPGQABK7K5XIDSU5IDRRCI5KXDGFRTNKJ7JQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.300312888 (553h53m49.171395923s) [noise=true , address=145.79.227.38:4977 , piece_id=QV2YZMQRL5ZR227NBTWCXF5DJSHHJ62MCCACZIQLYA7WB4VQ6BSQ, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:49.301792276 (553h53m49.172875321s) [noise=true , address=145.79.227.47:4265 , piece_id=Y772Y5ZFFDCHRRE5WAB4XNNL2TQ7WQCSGLBFBXHH5SSNG65JXJ2Q, seg_pos={0 0}, upload=485631]: Uploading piece... Dec 28 17:22:50.913044666 (553h53m50.784127711s) [upload=485631]: Next returned nil segment Dec 28 17:22:50.913056710 (553h53m50.784139745s) [upload=485631]: Waiting for error group managing segments... Dec 28 17:22:51.295865198 (553h53m51.166948243s) [noise=true , address=145.79.227.6:4258 , piece_id=YVJZARZ7B5BZYZYXEOTUCE2RQZVZRMYLZBPJR4IQGQUPSTUCOIMA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.296068094 (553h53m51.167151139s) [noise=true , address=145.79.227.11:4732 , piece_id=6MMYMMDHMKVAGT3WUOABDPFAFOSW4MKVP6QRXCHR5HX5NIB5FOMQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.303940945 (553h53m51.17502398s) [noise=true , address=145.79.227.38:4977 , piece_id=QV2YZMQRL5ZR227NBTWCXF5DJSHHJ62MCCACZIQLYA7WB4VQ6BSQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.313392253 (553h53m51.184475298s) [noise=true , address=145.79.227.50:4375 , piece_id=QDHDKKMNUJNIPISLFTNAGYOU4JZ34NWCQGAL2FABLTIYOSWYS57Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.317417217 (553h53m51.188500262s) [noise=true , address=145.79.227.3:4149 , piece_id=JPMIXVLL4L3ABJNN7TXMCKOGDRJVEQTL3G2FHJ3DVQB57OZEILOA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.329102457 (553h53m51.200185502s) [noise=true , address=145.79.227.39:4013 , piece_id=6622XCEZTSBLNR24JQ2T5VANM4PREH5YASZLHPRWQPOYDEAEPSYQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.336660270 (553h53m51.207743305s) [noise=true , address=145.79.227.14:4840 , piece_id=V5SMBB2BUBS3LKFJBJNSZJEUMMTAWJ5OQ7Y443THDR5AO6VAQ3UA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.338750239 (553h53m51.209833284s) [noise=true , address=145.79.227.47:4265 , piece_id=Y772Y5ZFFDCHRRE5WAB4XNNL2TQ7WQCSGLBFBXHH5SSNG65JXJ2Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.355191857 (553h53m51.226274902s) [noise=true , address=145.79.227.16:4910 , piece_id=BGLPWLKNLILD42VIFR6PHQAVWWEIOOAZI6EEMH3264ZUNFG2GGUQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.562580693 (553h53m51.433663728s) [noise=true , address=77.90.31.21:4878 , piece_id=TUJMMRDRXLQXLPU3BZZGTWNDM3WAH6JN2BYWRUAR7FUJ2MY3SSVA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:51.866792589 (553h53m51.737875624s) [noise=true , address=45.91.116.13:4154 , piece_id=URRSA6XX7YCXYDECIMSQ7I7FX2VFHMZBXKL3KHWQSSQP2OOH5LWQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.147275309 (553h53m52.018358344s) [noise=true , address=77.90.31.19:4842 , piece_id=Z4OCWJB4VIAPWCIBKPGQABK7K5XIDSU5IDRRCI5KXDGFRTNKJ7JQ, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.330287198 (553h53m52.201370253s) [noise=true , address=45.91.116.5:4297 , piece_id=5NXKOWBUM74XCJI5Q7SANUOZDRRE7L662FOIS5QK7RIDDLZXFCMA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: <nil> Dec 28 17:22:52.330307748 (553h53m52.201390783s) [seg_pos={0 0}, upload=485631]: Segment reached optimal threshold of 13 pieces. Dec 28 17:22:52.330496747 (553h53m52.201579802s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.330506065 (553h53m52.20158911s) [noise=true , address=77.90.31.13:4626 , piece_id=OFIS4QVZWJSM6M7OOUUV4746JGCTOKFUUDKSTSJEXCYIDGHK3F4Q, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332295342 (553h53m52.203378387s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.332311232 (553h53m52.203394267s) [noise=true , address=45.91.116.8:4045 , piece_id=KXTYJ2VCT52SUVEJUA23IX4AVF6SSS57M44HGKJACBKNOJZQFKBA, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332463042 (553h53m52.203546097s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:52.332471287 (553h53m52.203554332s) [noise=true , address=45.91.116.12:4227 , piece_id=5H3TGEJSDD6W4BZSYDB4KVP742Y3J475M6PS23PU5GSIQZ2G6TXA, seg_pos={0 0}, upload=485631]: Long tail cancellation due to optimal threshold Dec 28 17:22:52.332489581 (553h53m52.203572627s) [seg_pos={0 0}, upload=485631]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:22:52.332568272 (553h53m52.203651317s) [upload=485631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:22:52.332574925 (553h53m52.20365796s) [upload=485631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:22:52.615749100 (553h53m52.486832135s) [upload=485631]: Done uploading segments... Dec 28 17:22:52.615766212 (553h53m52.486849248s) [upload=485631]: Upload finished. err: <nil> Duration: 30.004523788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X........................X.............................] duration: 46.276562229s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 46.276582446s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X.XXXX.X.X....X.....XXX.X.......X..X....................X..X] duration: 5.001934928s trace: https://jaeger.dev.storj.io/trace/4491e66b71cf4d0a FAIL (2025-12-28 17:22:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001956418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX....X.X.XXX..X.X...XX...XX..XX.........XXX...XX...X.X..XX] duration: 30.00306115s trace: https://jaeger.dev.storj.io/trace/64f16a464f007587 FAIL (2025-12-28 17:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: DAAD2NJZKN6SMFJN26QRZ4K7EUQKEQIYES6DPJ3U5PAZVB4CIWQQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: W2MEFXP5LYH32RRUJ6QF6BT6HDTXKMM2S7VLW6VWJUXBPJCOKI4A) context deadline exceeded ======LOGS====== Dec 28 17:21:58.800970986 (553h52m58.672054021s) [upload=485619]: Starting upload Dec 28 17:21:58.801071127 (553h52m58.672154162s) [upload=485619]: Done starting upload Dec 28 17:21:58.801103378 (553h52m58.672186423s) [upload=485619]: Uploading segments... Dec 28 17:21:58.801321494 (553h52m58.672404529s) [upload=485619]: Got next segment. Inline: false Dec 28 17:21:58.801367982 (553h52m58.672451017s) [upload=485619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 17:21:58.801374063 (553h52m58.672457118s) [upload=485619]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 17:21:59.046951569 (553h52m58.918034604s) [seg_pos={0 0}, upload=485619]: Begin upload segment... Dec 28 17:21:59.047000511 (553h52m58.918083546s) [seg_pos={0 0}, upload=485619]: Done begin upload segment. Dec 28 17:21:59.047039195 (553h52m58.91812226s) [noise=true , address=77.90.31.18:4805 , piece_id=DAAD2NJZKN6SMFJN26QRZ4K7EUQKEQIYES6DPJ3U5PAZVB4CIWQQ, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.047121853 (553h52m58.918204898s) [noise=true , address=145.79.227.45:4229 , piece_id=FQWV7MT4R3KFDC5WFTZM5MTZJCCZTJHYUIDTKN7NJTC2KUFQTAXA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.047790075 (553h52m58.91887313s) [noise=true , address=145.79.227.2:4115 , piece_id=INFX52NERB73JA354O2OYXCWBLMDITJE4BTQJCT7C2U6UEUEQARA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.048278625 (553h52m58.91936166s) [noise=true , address=145.79.227.37:4565 , piece_id=T3EOZMFUJEXGM2USIAAHUQDZCA7AF5J3QHXVKZGMRVAIRZQAOYYA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.048663027 (553h52m58.919746072s) [noise=true , address=145.79.227.46:4602 , piece_id=RSBD5WETCH5GQQM4CV6WXILQF35MRGIR26GVFNUBKK5IKS77W5GQ, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.049048151 (553h52m58.920131196s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSHWLTRJSKLGSZPXYYE55DKEN2QKJTIHFD5VHGONVCAK3G3UQX5Q, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.049353222 (553h52m58.920436257s) [noise=true , address=45.91.116.13:4155 , piece_id=W2MEFXP5LYH32RRUJ6QF6BT6HDTXKMM2S7VLW6VWJUXBPJCOKI4A, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.049655827 (553h52m58.920738872s) [noise=true , address=145.79.227.41:4085 , piece_id=A6EDIUH4RSXAEOB2IW6N2MKF6V2GBDEEXPP2KKZF7BAJKWWKXISA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.050199463 (553h52m58.921282528s) [noise=true , address=77.90.31.16:4736 , piece_id=NJJVU2OI2MEGGZOSHQKB3OSWX4YYNBVIPZHVVA777RHYLNYUYTXA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.050744641 (553h52m58.921827676s) [noise=true , address=45.91.116.5:4297 , piece_id=42FDJTEW5ILTYSEIMP3CYHHXZ6B7IRJ6UP2GEMZEWG6NQDNRAM3Q, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.051098896 (553h52m58.922181931s) [noise=true , address=145.79.227.17:4971 , piece_id=YO63BHPTXBOHJUGHENDYOEFRAJOX2HJDWDH7MPBE7V7DHQCKC2ZQ, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.051437381 (553h52m58.922520426s) [noise=true , address=145.79.227.10:4401 , piece_id=AFAUQDVMB7M34QN2A44M2TC5VNBMPL7PZU4OWDPBMZWKKVUJLL5A, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.051577738 (553h52m58.922660773s) [noise=true , address=145.79.227.19:4079 , piece_id=JXESI4TJELNBKIDCEB2UYURKNQZXQFP4ARLVRUUVGGMWYQCLKO4A, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.051847171 (553h52m58.922930216s) [noise=true , address=145.79.227.22:4509 , piece_id=YCBGVBEBMHS2OYSLSWMBESJVVNZBU3LHWBCYFCNDA2UU2T333QDA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.052102166 (553h52m58.923185211s) [noise=true , address=145.79.227.15:4874 , piece_id=NCHZ46MIZQGL6D7SE7MHIRCIOIEKQJO4JOBJTQAIKRYK45R2XJZA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:21:59.052681059 (553h52m58.923764104s) [noise=true , address=145.79.227.38:4977 , piece_id=X6XRAP4JBFIZIDTR4KV326UU4Z5JDZLQL6FMXK7FV7UUISHHELNA, seg_pos={0 0}, upload=485619]: Uploading piece... Dec 28 17:22:00.754329781 (553h53m0.625412826s) [upload=485619]: Next returned nil segment Dec 28 17:22:00.754346764 (553h53m0.625429809s) [upload=485619]: Waiting for error group managing segments... Dec 28 17:22:01.042404108 (553h53m0.913487153s) [noise=true , address=145.79.227.41:4085 , piece_id=A6EDIUH4RSXAEOB2IW6N2MKF6V2GBDEEXPP2KKZF7BAJKWWKXISA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.055461952 (553h53m0.926544997s) [noise=true , address=145.79.227.10:4401 , piece_id=AFAUQDVMB7M34QN2A44M2TC5VNBMPL7PZU4OWDPBMZWKKVUJLL5A, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.056247598 (553h53m0.927330644s) [noise=true , address=145.79.227.19:4079 , piece_id=JXESI4TJELNBKIDCEB2UYURKNQZXQFP4ARLVRUUVGGMWYQCLKO4A, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.056794479 (553h53m0.927877524s) [noise=true , address=145.79.227.38:4977 , piece_id=X6XRAP4JBFIZIDTR4KV326UU4Z5JDZLQL6FMXK7FV7UUISHHELNA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.057276237 (553h53m0.928359282s) [noise=true , address=145.79.227.46:4602 , piece_id=RSBD5WETCH5GQQM4CV6WXILQF35MRGIR26GVFNUBKK5IKS77W5GQ, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.058935306 (553h53m0.930018351s) [noise=true , address=145.79.227.37:4565 , piece_id=T3EOZMFUJEXGM2USIAAHUQDZCA7AF5J3QHXVKZGMRVAIRZQAOYYA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.061448271 (553h53m0.932531316s) [noise=true , address=145.79.227.45:4229 , piece_id=FQWV7MT4R3KFDC5WFTZM5MTZJCCZTJHYUIDTKN7NJTC2KUFQTAXA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.067005533 (553h53m0.938088568s) [noise=true , address=145.79.227.22:4509 , piece_id=YCBGVBEBMHS2OYSLSWMBESJVVNZBU3LHWBCYFCNDA2UU2T333QDA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.067462213 (553h53m0.938545258s) [noise=true , address=145.79.227.17:4971 , piece_id=YO63BHPTXBOHJUGHENDYOEFRAJOX2HJDWDH7MPBE7V7DHQCKC2ZQ, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.079144017 (553h53m0.950227063s) [noise=true , address=145.79.227.15:4874 , piece_id=NCHZ46MIZQGL6D7SE7MHIRCIOIEKQJO4JOBJTQAIKRYK45R2XJZA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.242534402 (553h53m1.113617447s) [noise=true , address=77.90.31.16:4736 , piece_id=NJJVU2OI2MEGGZOSHQKB3OSWX4YYNBVIPZHVVA777RHYLNYUYTXA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.247085929 (553h53m1.118168975s) [noise=true , address=77.90.31.18:4805 , piece_id=DAAD2NJZKN6SMFJN26QRZ4K7EUQKEQIYES6DPJ3U5PAZVB4CIWQQ, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.292401119 (553h53m1.163484184s) [noise=true , address=45.91.116.13:4155 , piece_id=W2MEFXP5LYH32RRUJ6QF6BT6HDTXKMM2S7VLW6VWJUXBPJCOKI4A, seg_pos={0 0}, upload=485619]: Done uploading piece. err: <nil> Dec 28 17:22:01.292428861 (553h53m1.163511906s) [seg_pos={0 0}, upload=485619]: Segment reached optimal threshold of 13 pieces. Dec 28 17:22:01.292675941 (553h53m1.163758986s) [noise=true , address=45.91.116.5:4297 , piece_id=42FDJTEW5ILTYSEIMP3CYHHXZ6B7IRJ6UP2GEMZEWG6NQDNRAM3Q, seg_pos={0 0}, upload=485619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:01.292685370 (553h53m1.163768415s) [noise=true , address=45.91.116.5:4297 , piece_id=42FDJTEW5ILTYSEIMP3CYHHXZ6B7IRJ6UP2GEMZEWG6NQDNRAM3Q, seg_pos={0 0}, upload=485619]: Long tail cancellation due to optimal threshold Dec 28 17:22:01.292785751 (553h53m1.163868826s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSHWLTRJSKLGSZPXYYE55DKEN2QKJTIHFD5VHGONVCAK3G3UQX5Q, seg_pos={0 0}, upload=485619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:01.292822671 (553h53m1.163905736s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSHWLTRJSKLGSZPXYYE55DKEN2QKJTIHFD5VHGONVCAK3G3UQX5Q, seg_pos={0 0}, upload=485619]: Long tail cancellation due to optimal threshold Dec 28 17:22:01.293072497 (553h53m1.164155572s) [noise=true , address=145.79.227.2:4115 , piece_id=INFX52NERB73JA354O2OYXCWBLMDITJE4BTQJCT7C2U6UEUEQARA, seg_pos={0 0}, upload=485619]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 28 17:22:01.293082455 (553h53m1.164165501s) [noise=true , address=145.79.227.2:4115 , piece_id=INFX52NERB73JA354O2OYXCWBLMDITJE4BTQJCT7C2U6UEUEQARA, seg_pos={0 0}, upload=485619]: Long tail cancellation due to optimal threshold Dec 28 17:22:01.293107594 (553h53m1.164190639s) [seg_pos={0 0}, upload=485619]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 17:22:01.293193407 (553h53m1.164276452s) [upload=485619]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 17:22:01.293200530 (553h53m1.164283585s) [upload=485619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 17:22:01.591508765 (553h53m1.4625918s) [upload=485619]: Done uploading segments... Dec 28 17:22:01.591530385 (553h53m1.46261342s) [upload=485619]: Upload finished. err: <nil> Duration: 30.003072192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X........................X............................] duration: 38.768247876s trace: https://jaeger.dev.storj.io/trace/15189525548382b4 FAIL (2025-12-28 17:22:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 16:56:36.773317205 (553h27m36.64440025s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.773373252 (553h27m36.644456298s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774100617 (553h27m36.645183662s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774483165 (553h27m36.6455662s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.774867687 (553h27m36.645950722s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775233324 (553h27m36.646316369s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775612566 (553h27m36.646695611s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.775984865 (553h27m36.64706791s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.778185395 (553h27m36.64926843s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.779727722 (553h27m36.650810757s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.781320876 (553h27m36.652403921s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.783247795 (553h27m36.65433084s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.785987642 (553h27m36.657070677s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.787823719 (553h27m36.658906754s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.789193426 (553h27m36.660276461s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.790419851 (553h27m36.661502886s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:36.791315076 (553h27m36.662398121s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Uploading piece... Dec 28 16:56:38.255642428 (553h27m38.126725473s) [upload=485240]: Next returned nil segment Dec 28 16:56:38.255656174 (553h27m38.126739209s) [upload=485240]: Waiting for error group managing segments... Dec 28 16:56:38.685483761 (553h27m38.556566806s) [noise=true , address=145.79.227.5:4227 , piece_id=WTIBG76U5SNRTNGCNAXCVZW7VHHSVF7VQKOUBTMRD6FL6DEYG7WA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.694641700 (553h27m38.565724735s) [noise=true , address=145.79.227.29:4971 , piece_id=LXHJRU2FTSG4TURLDBVU43LIXCGJODB5JL3UDZLCHX4Q2AJM5NUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.823845330 (553h27m38.694928365s) [noise=true , address=77.90.31.14:4674 , piece_id=EKAJVUD25CZMU7NJURLTIHRMKHEKRPWI2HFQDDRVGEAKH5757ORQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.907605256 (553h27m38.778688291s) [noise=true , address=77.90.31.11:4588 , piece_id=KX7563Q4NL3EL7BVRRFK7ZTB4Q2JQOEJ37T2JSS67GLKKO6XXL6Q, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.918588950 (553h27m38.789671985s) [noise=true , address=77.90.31.13:4635 , piece_id=5CFHWMPLCTTEKFB4LP7QPFNPUPR7WWWSINJUIEPFNP44Z67KPWHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:38.937055104 (553h27m38.808138149s) [noise=true , address=77.90.31.16:4739 , piece_id=QSOPYXB7ZY3227KUY2SHUWA2NCKKSBNTEI7FO44POQT5KSS3PTQA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.327560956 (553h27m39.198643991s) [noise=true , address=64.49.13.137:4502 , piece_id=5IS2LGRMV2EUTJQ2NWLXZX6WXOYIO7EYZBQWJ7DAX3R42P6UAVBA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.499062238 (553h27m39.370145284s) [noise=true , address=77.90.31.10:4538 , piece_id=VF5Q7IBBSODVRD6MGWDSYAUWM7ZWQZQBXBZGOONYXBZUKQ2G6ZUQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.514043335 (553h27m39.38512637s) [noise=true , address=77.90.31.15:4711 , piece_id=BTCZ6SA3V7D7TAM4SWWYUKPGOSSHVFBB2S5G7VMJNFIWQPQ7LKOA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.591080830 (553h27m39.462163875s) [noise=true , address=145.79.227.40:4057 , piece_id=QN3PSGD7HCJ63MIHMU5BKDPILQZ37HXMFI5FC6CMFEDJAILDTIHA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.600966605 (553h27m39.47204964s) [noise=true , address=77.90.31.18:4822 , piece_id=DTPBTIUTKPJT35QOWZCHDYSSZ5KWONJY66OEEDAMYTMYQTUMZVAQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.623085073 (553h27m39.494168108s) [noise=true , address=145.79.227.30:4775 , piece_id=ER4TAKEGOLTOSINTEKQNN6Y7QEC2E6JHIT33MLYKR2CWASA3ZKLQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.643856074 (553h27m39.51493912s) [noise=true , address=77.90.31.17:4780 , piece_id=XTTNABT6R22CAXCUCU5V5GFS5O7EDUYAOY5J7PZCYIJCOJQLOTYA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753021828 (553h27m39.624104863s) [noise=true , address=145.79.227.49:4351 , piece_id=QD42KYDMYFBUB2JE5LI2T3YIQPNRWOJIYTKHJ2IUO2AC4QRQQXKA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: <nil> Dec 28 16:56:39.753047307 (553h27m39.624130352s) [seg_pos={0 0}, upload=485240]: Segment reached optimal threshold of 14 pieces. Dec 28 16:56:39.753290259 (553h27m39.624373294s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753301480 (553h27m39.624384525s) [noise=true , address=145.79.227.35:4510 , piece_id=OV3GL434ENGBVTUQTTV457YC4DMMY4V37NP4MBJO4Q6GUW3QXPCA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753426639 (553h27m39.624509674s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pXfQm637dESPkrLBWSNALryxEwooBFdQCh1oyB4emUFes2pvx): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753458480 (553h27m39.624541515s) [noise=true , address=145.79.227.17:4990 , piece_id=3QIAUJEOXY4NWMIEMGIAI7V6LQBTFRVGG7U753BJR3XBHYIEIGMQ, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753473177 (553h27m39.624556222s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kmvJbEBDVsuL47p1Nu8qMxQtHKGiBWH4F4BRYaBrXqrz7ZRRV): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753483047 (553h27m39.624566102s) [noise=true , address=45.91.116.9:4282 , piece_id=TDMDPAPQWRFQGYMRVCBFHD2CSZLXIUBJBVOI3PDTS6NFNBQUR7BA, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753653300 (553h27m39.624736355s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 28 16:56:39.753663761 (553h27m39.624746796s) [noise=true , address=193.149.164.143:4704 , piece_id=MYKKFRSJFA4TRHLWUADY7LTUAQ223HIPSM7LH5OHGGWNQYVMOK3A, seg_pos={0 0}, upload=485240]: Long tail cancellation due to optimal threshold Dec 28 16:56:39.753731079 (553h27m39.624814114s) [seg_pos={0 0}, upload=485240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:56:39.753993819 (553h27m39.625076864s) [upload=485240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:56:39.754048053 (553h27m39.625131098s) [upload=485240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:56:40.059228537 (553h27m39.930311572s) [upload=485240]: Done uploading segments... Dec 28 16:56:40.059253655 (553h27m39.9303367s) [upload=485240]: Upload finished. err: <nil> Duration: 38.76826516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X.XXXX.X.X....X.....XXX.X.......X..X....................X..] duration: 3.541177546s trace: https://jaeger.dev.storj.io/trace/471abe2d6507aac8 FAIL (2025-12-28 17:22:16 UTC) orbiter warning: context deadline exceeded Duration: 3.541207763s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)