Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X.XXX.] duration: 10.44857171s trace: https://jaeger.dev.storj.io/trace/7b73edf406cbb359 FAIL (2025-12-27 01:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:58:00.012658171 (514h28m59.883741206s) [upload=451460]: Starting upload Dec 27 01:58:00.012713357 (514h28m59.883796392s) [upload=451460]: Done starting upload Dec 27 01:58:00.012908509 (514h28m59.883991554s) [upload=451460]: Uploading segments... Dec 27 01:58:00.013020953 (514h28m59.884103988s) [upload=451460]: Got next segment. Inline: false Dec 27 01:58:00.013075736 (514h28m59.884158781s) [upload=451460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:58:00.013082048 (514h28m59.884165093s) [upload=451460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:58:00.260276893 (514h29m0.131359948s) [seg_pos={0 0}, upload=451460]: Begin upload segment... Dec 27 01:58:00.260340284 (514h29m0.131423319s) [seg_pos={0 0}, upload=451460]: Done begin upload segment. Dec 27 01:58:00.260386132 (514h29m0.131469167s) [noise=true , address=145.79.227.17:4971 , piece_id=JJHZCY7WKWNFDSLF46ZD3KVNADCX5BWWV56OS6XX3MY24RYESYDQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260392994 (514h29m0.131476029s) [noise=true , address=145.79.227.38:4978 , piece_id=VVHKMJTSQQACJI6YZX5TBDCBKBJ6AVRXBXDYUIYCC7EQHSTX4ORA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260828513 (514h29m0.131911558s) [noise=true , address=77.90.31.21:4878 , piece_id=KW7TSB4XUHJX6DUQPGZV45OQ7LD6JSCIRG3BA7AVYMXYKDVRBBQA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260976375 (514h29m0.132059411s) [noise=true , address=145.79.227.51:4411 , piece_id=YWKRQCF72M3ETXTGX4ADETURVVAYEGS7OOZTBMB2CSU4ZE352FDQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261257992 (514h29m0.132341037s) [noise=true , address=45.91.116.10:4081 , piece_id=T62N4CY3BWQPA6PW65GJOQ6FQVO2RNINFOARQXJ7T7HTTEBI5KPQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261267399 (514h29m0.132350444s) [noise=true , address=145.79.227.15:4875 , piece_id=PHWNPYQGG5A2IQ36WEC6JJIMTD6CPRWX5H72NT3QVFKZOF3RVMRA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261621834 (514h29m0.132704879s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261634438 (514h29m0.132717473s) [noise=true , address=145.79.227.42:4123 , piece_id=TONW3MIGSHEXOJOFPJMS75OBQNXTGHRBXLJU62SNROZ7YADYY6LQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261917538 (514h29m0.133000573s) [noise=true , address=145.79.227.12:4765 , piece_id=7XSPALGNPWA7HIH5WVAN5DYJ3IUJSNZZWEZNG634E7TVRYW2OH6Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262002429 (514h29m0.133085464s) [noise=true , address=145.79.227.19:4077 , piece_id=OV5MQM64YODKB2IVGIKBRCRAI54WWZXCXJKBSULZFTNRIEJP2Z7Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262200286 (514h29m0.133283321s) [noise=true , address=145.79.227.50:4373 , piece_id=FKHZOEXGBDU6YYFCFQRNIFL4GUCD3Q7ZLFH5BEGYWCQYL6H7I4OA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262305105 (514h29m0.133388141s) [noise=true , address=45.91.116.6:4974 , piece_id=PSUTNXPGE423G3AMFOKZ2VWI47B6KLAHMXBO46CXRBXBLVCPBYHA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262600067 (514h29m0.133683112s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262917671 (514h29m0.134000706s) [noise=true , address=145.79.227.37:4565 , piece_id=UJAENTVZRIZAOPCCTPCJQG6WJNBJSW5WOKFL4LDQIAXQH66QO22A, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.263587227 (514h29m0.134670262s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.263988200 (514h29m0.135071245s) [noise=true , address=145.79.227.39:4013 , piece_id=5NOQGLKNGGTRKJQMZT7KMAFTDUXCT567DBY2XZ2KWFPTQRTOCXYA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:01.964916803 (514h29m1.835999838s) [upload=451460]: Next returned nil segment Dec 27 01:58:01.964935429 (514h29m1.836018494s) [upload=451460]: Waiting for error group managing segments... Dec 27 01:58:02.258688466 (514h29m2.129771502s) [noise=true , address=145.79.227.15:4875 , piece_id=PHWNPYQGG5A2IQ36WEC6JJIMTD6CPRWX5H72NT3QVFKZOF3RVMRA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.261036597 (514h29m2.132119642s) [noise=true , address=145.79.227.37:4565 , piece_id=UJAENTVZRIZAOPCCTPCJQG6WJNBJSW5WOKFL4LDQIAXQH66QO22A, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.261227862 (514h29m2.132310907s) [noise=true , address=145.79.227.51:4411 , piece_id=YWKRQCF72M3ETXTGX4ADETURVVAYEGS7OOZTBMB2CSU4ZE352FDQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.271655057 (514h29m2.142738102s) [noise=true , address=145.79.227.38:4978 , piece_id=VVHKMJTSQQACJI6YZX5TBDCBKBJ6AVRXBXDYUIYCC7EQHSTX4ORA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.271747733 (514h29m2.142830779s) [noise=true , address=145.79.227.39:4013 , piece_id=5NOQGLKNGGTRKJQMZT7KMAFTDUXCT567DBY2XZ2KWFPTQRTOCXYA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.275365122 (514h29m2.146448167s) [noise=true , address=145.79.227.42:4123 , piece_id=TONW3MIGSHEXOJOFPJMS75OBQNXTGHRBXLJU62SNROZ7YADYY6LQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.291965541 (514h29m2.163048596s) [noise=true , address=145.79.227.19:4077 , piece_id=OV5MQM64YODKB2IVGIKBRCRAI54WWZXCXJKBSULZFTNRIEJP2Z7Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.294061602 (514h29m2.165144647s) [noise=true , address=145.79.227.12:4765 , piece_id=7XSPALGNPWA7HIH5WVAN5DYJ3IUJSNZZWEZNG634E7TVRYW2OH6Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.296832358 (514h29m2.167915393s) [noise=true , address=145.79.227.50:4373 , piece_id=FKHZOEXGBDU6YYFCFQRNIFL4GUCD3Q7ZLFH5BEGYWCQYL6H7I4OA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.503419279 (514h29m2.374502324s) [noise=true , address=45.91.116.10:4081 , piece_id=T62N4CY3BWQPA6PW65GJOQ6FQVO2RNINFOARQXJ7T7HTTEBI5KPQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.523578746 (514h29m2.394661791s) [noise=true , address=45.91.116.6:4974 , piece_id=PSUTNXPGE423G3AMFOKZ2VWI47B6KLAHMXBO46CXRBXBLVCPBYHA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.525689736 (514h29m2.396772781s) [noise=true , address=77.90.31.21:4878 , piece_id=KW7TSB4XUHJX6DUQPGZV45OQ7LD6JSCIRG3BA7AVYMXYKDVRBBQA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.590005752 (514h29m2.461088797s) [noise=true , address=145.79.227.17:4971 , piece_id=JJHZCY7WKWNFDSLF46ZD3KVNADCX5BWWV56OS6XX3MY24RYESYDQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.590031371 (514h29m2.461114416s) [seg_pos={0 0}, upload=451460]: Segment reached optimal threshold of 13 pieces. Dec 27 01:58:02.590262742 (514h29m2.461345797s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 01:58:02.590273051 (514h29m2.461356097s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590567803 (514h29m2.461650868s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection Dec 27 01:58:02.590577031 (514h29m2.461660076s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590712689 (514h29m2.461795734s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 01:58:02.590734229 (514h29m2.461817274s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590769146 (514h29m2.461852181s) [seg_pos={0 0}, upload=451460]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:58:02.590880518 (514h29m2.461963553s) [upload=451460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:58:02.590888083 (514h29m2.461971128s) [upload=451460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:58:02.875395575 (514h29m2.74647862s) [upload=451460]: Done uploading segments... Dec 27 01:58:02.875417047 (514h29m2.746500082s) [upload=451460]: Upload finished. err: <nil> Duration: 10.448598142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X.XXX] duration: 30.002298607s trace: https://jaeger.dev.storj.io/trace/7b73edf406cbb359 FAIL (2025-12-27 01:58:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:58:00.012658171 (514h28m59.883741206s) [upload=451460]: Starting upload Dec 27 01:58:00.012713357 (514h28m59.883796392s) [upload=451460]: Done starting upload Dec 27 01:58:00.012908509 (514h28m59.883991554s) [upload=451460]: Uploading segments... Dec 27 01:58:00.013020953 (514h28m59.884103988s) [upload=451460]: Got next segment. Inline: false Dec 27 01:58:00.013075736 (514h28m59.884158781s) [upload=451460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:58:00.013082048 (514h28m59.884165093s) [upload=451460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:58:00.260276893 (514h29m0.131359948s) [seg_pos={0 0}, upload=451460]: Begin upload segment... Dec 27 01:58:00.260340284 (514h29m0.131423319s) [seg_pos={0 0}, upload=451460]: Done begin upload segment. Dec 27 01:58:00.260386132 (514h29m0.131469167s) [noise=true , address=145.79.227.17:4971 , piece_id=JJHZCY7WKWNFDSLF46ZD3KVNADCX5BWWV56OS6XX3MY24RYESYDQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260392994 (514h29m0.131476029s) [noise=true , address=145.79.227.38:4978 , piece_id=VVHKMJTSQQACJI6YZX5TBDCBKBJ6AVRXBXDYUIYCC7EQHSTX4ORA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260828513 (514h29m0.131911558s) [noise=true , address=77.90.31.21:4878 , piece_id=KW7TSB4XUHJX6DUQPGZV45OQ7LD6JSCIRG3BA7AVYMXYKDVRBBQA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.260976375 (514h29m0.132059411s) [noise=true , address=145.79.227.51:4411 , piece_id=YWKRQCF72M3ETXTGX4ADETURVVAYEGS7OOZTBMB2CSU4ZE352FDQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261257992 (514h29m0.132341037s) [noise=true , address=45.91.116.10:4081 , piece_id=T62N4CY3BWQPA6PW65GJOQ6FQVO2RNINFOARQXJ7T7HTTEBI5KPQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261267399 (514h29m0.132350444s) [noise=true , address=145.79.227.15:4875 , piece_id=PHWNPYQGG5A2IQ36WEC6JJIMTD6CPRWX5H72NT3QVFKZOF3RVMRA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261621834 (514h29m0.132704879s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261634438 (514h29m0.132717473s) [noise=true , address=145.79.227.42:4123 , piece_id=TONW3MIGSHEXOJOFPJMS75OBQNXTGHRBXLJU62SNROZ7YADYY6LQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.261917538 (514h29m0.133000573s) [noise=true , address=145.79.227.12:4765 , piece_id=7XSPALGNPWA7HIH5WVAN5DYJ3IUJSNZZWEZNG634E7TVRYW2OH6Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262002429 (514h29m0.133085464s) [noise=true , address=145.79.227.19:4077 , piece_id=OV5MQM64YODKB2IVGIKBRCRAI54WWZXCXJKBSULZFTNRIEJP2Z7Q, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262200286 (514h29m0.133283321s) [noise=true , address=145.79.227.50:4373 , piece_id=FKHZOEXGBDU6YYFCFQRNIFL4GUCD3Q7ZLFH5BEGYWCQYL6H7I4OA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262305105 (514h29m0.133388141s) [noise=true , address=45.91.116.6:4974 , piece_id=PSUTNXPGE423G3AMFOKZ2VWI47B6KLAHMXBO46CXRBXBLVCPBYHA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262600067 (514h29m0.133683112s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.262917671 (514h29m0.134000706s) [noise=true , address=145.79.227.37:4565 , piece_id=UJAENTVZRIZAOPCCTPCJQG6WJNBJSW5WOKFL4LDQIAXQH66QO22A, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.263587227 (514h29m0.134670262s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:00.263988200 (514h29m0.135071245s) [noise=true , address=145.79.227.39:4013 , piece_id=5NOQGLKNGGTRKJQMZT7KMAFTDUXCT567DBY2XZ2KWFPTQRTOCXYA, seg_pos={0 0}, upload=451460]: Uploading piece... Dec 27 01:58:01.964916803 (514h29m1.835999838s) [upload=451460]: Next returned nil segment Dec 27 01:58:01.964935429 (514h29m1.836018494s) [upload=451460]: Waiting for error group managing segments... Dec 27 01:58:02.258688466 (514h29m2.129771502s) [noise=true , address=145.79.227.15:4875 , piece_id=PHWNPYQGG5A2IQ36WEC6JJIMTD6CPRWX5H72NT3QVFKZOF3RVMRA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.261036597 (514h29m2.132119642s) [noise=true , address=145.79.227.37:4565 , piece_id=UJAENTVZRIZAOPCCTPCJQG6WJNBJSW5WOKFL4LDQIAXQH66QO22A, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.261227862 (514h29m2.132310907s) [noise=true , address=145.79.227.51:4411 , piece_id=YWKRQCF72M3ETXTGX4ADETURVVAYEGS7OOZTBMB2CSU4ZE352FDQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.271655057 (514h29m2.142738102s) [noise=true , address=145.79.227.38:4978 , piece_id=VVHKMJTSQQACJI6YZX5TBDCBKBJ6AVRXBXDYUIYCC7EQHSTX4ORA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.271747733 (514h29m2.142830779s) [noise=true , address=145.79.227.39:4013 , piece_id=5NOQGLKNGGTRKJQMZT7KMAFTDUXCT567DBY2XZ2KWFPTQRTOCXYA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.275365122 (514h29m2.146448167s) [noise=true , address=145.79.227.42:4123 , piece_id=TONW3MIGSHEXOJOFPJMS75OBQNXTGHRBXLJU62SNROZ7YADYY6LQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.291965541 (514h29m2.163048596s) [noise=true , address=145.79.227.19:4077 , piece_id=OV5MQM64YODKB2IVGIKBRCRAI54WWZXCXJKBSULZFTNRIEJP2Z7Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.294061602 (514h29m2.165144647s) [noise=true , address=145.79.227.12:4765 , piece_id=7XSPALGNPWA7HIH5WVAN5DYJ3IUJSNZZWEZNG634E7TVRYW2OH6Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.296832358 (514h29m2.167915393s) [noise=true , address=145.79.227.50:4373 , piece_id=FKHZOEXGBDU6YYFCFQRNIFL4GUCD3Q7ZLFH5BEGYWCQYL6H7I4OA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.503419279 (514h29m2.374502324s) [noise=true , address=45.91.116.10:4081 , piece_id=T62N4CY3BWQPA6PW65GJOQ6FQVO2RNINFOARQXJ7T7HTTEBI5KPQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.523578746 (514h29m2.394661791s) [noise=true , address=45.91.116.6:4974 , piece_id=PSUTNXPGE423G3AMFOKZ2VWI47B6KLAHMXBO46CXRBXBLVCPBYHA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.525689736 (514h29m2.396772781s) [noise=true , address=77.90.31.21:4878 , piece_id=KW7TSB4XUHJX6DUQPGZV45OQ7LD6JSCIRG3BA7AVYMXYKDVRBBQA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.590005752 (514h29m2.461088797s) [noise=true , address=145.79.227.17:4971 , piece_id=JJHZCY7WKWNFDSLF46ZD3KVNADCX5BWWV56OS6XX3MY24RYESYDQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: <nil> Dec 27 01:58:02.590031371 (514h29m2.461114416s) [seg_pos={0 0}, upload=451460]: Segment reached optimal threshold of 13 pieces. Dec 27 01:58:02.590262742 (514h29m2.461345797s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 01:58:02.590273051 (514h29m2.461356097s) [noise=true , address=145.79.227.40:4051 , piece_id=QCBJBYGORNFCF6VLDCR3EAO32BZLRKQF2BAXZM672ERRN2NGLDMQ, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590567803 (514h29m2.461650868s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42794->145.79.227.3:4151: use of closed network connection Dec 27 01:58:02.590577031 (514h29m2.461660076s) [noise=true , address=145.79.227.3:4151 , piece_id=U55RXWRVI2MQZZ7AKPMMKIVR5OWW4CN54DOV736HZ3R36F44XB4Q, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590712689 (514h29m2.461795734s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 01:58:02.590734229 (514h29m2.461817274s) [noise=true , address=45.91.116.12:4228 , piece_id=BKOS7POQQBOPFT43VN7PZQ2B644FI74U2QMPCIUTDZPNJV7TIAOA, seg_pos={0 0}, upload=451460]: Long tail cancellation due to optimal threshold Dec 27 01:58:02.590769146 (514h29m2.461852181s) [seg_pos={0 0}, upload=451460]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:58:02.590880518 (514h29m2.461963553s) [upload=451460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:58:02.590888083 (514h29m2.461971128s) [upload=451460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:58:02.875395575 (514h29m2.74647862s) [upload=451460]: Done uploading segments... Dec 27 01:58:02.875417047 (514h29m2.746500082s) [upload=451460]: Upload finished. err: <nil> Duration: 30.002323497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X.XX] duration: 30.002291937s trace: https://jaeger.dev.storj.io/trace/151fe598eb9b70a6 FAIL (2025-12-27 01:57:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: P2OHQJV53WW6NPAQK6KY67U4UASIQYGFJERMAUP5HJHJHXUB34QA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SFDZ5ATRSW4MWAARFOTJD3APEAIY3MNJIVKHQBZA3A4KUFCGNKQA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: ZHR2SC4QGEV42TJEGHD2G3D67XNTJWPAXM56BA5YBEWTTOZEW4FA) context deadline exceeded ======LOGS====== 27 01:56:45.538386130 (514h27m45.409469165s) [upload=451446]: Starting upload Dec 27 01:56:45.538573878 (514h27m45.409656954s) [upload=451446]: Done starting upload Dec 27 01:56:45.538669289 (514h27m45.409752355s) [upload=451446]: Uploading segments... Dec 27 01:56:45.539038583 (514h27m45.410121628s) [upload=451446]: Got next segment. Inline: false Dec 27 01:56:45.539079370 (514h27m45.410162416s) [upload=451446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:56:45.539085231 (514h27m45.410168267s) [upload=451446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:56:45.790372820 (514h27m45.661455855s) [seg_pos={0 0}, upload=451446]: Begin upload segment... Dec 27 01:56:45.790425349 (514h27m45.661508384s) [seg_pos={0 0}, upload=451446]: Done begin upload segment. Dec 27 01:56:45.790580034 (514h27m45.661663069s) [noise=true , address=145.79.227.16:4909 , piece_id=QAEBH4FSG2TCSHDSPQSRBQ5LEM3KLLAJYVKOADLYHECLUXL2S3UQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.790628035 (514h27m45.66171107s) [noise=true , address=145.79.227.49:4339 , piece_id=B4V37DEQHYZ6FIK7FLN3LWNP6444GXFJ3XB7QMUJCVQEFC3VRNWQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.791251332 (514h27m45.662334367s) [noise=true , address=77.90.31.17:4773 , piece_id=BWRG6C4SAA7VX2YDLUWUHIHUS6FF32BERP3GVQAKRSB56G7OKUCQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.791650131 (514h27m45.662733166s) [noise=true , address=77.90.31.11:4556 , piece_id=P2OHQJV53WW6NPAQK6KY67U4UASIQYGFJERMAUP5HJHJHXUB34QA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.792015297 (514h27m45.663098342s) [noise=true , address=145.79.227.6:4259 , piece_id=AC5TBXJGL6QKBJPYJWN23ANO4OQNCNZSVLY3PEGC5AE3R57FCZXA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.792438773 (514h27m45.663521818s) [noise=true , address=145.79.227.37:4566 , piece_id=VOY4BMWP7PCPPWOVT6WCVDCGATXHPVNN6VYSHHXPAY5LR23WHLIQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.792517814 (514h27m45.663600849s) [noise=true , address=145.79.227.2:4115 , piece_id=TK7BAFOAVZ5RYH5ICFOYRO4OMMOFSJFDHBI7WEPISV6ZR77KDB4A, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.793019810 (514h27m45.664102855s) [noise=true , address=77.90.31.14:4664 , piece_id=FVSJPKN6SSW35QILLBUUZNKC7RTRVEPBSCQJ6VDTJWNIRFLPOOQA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.793591538 (514h27m45.664674573s) [noise=true , address=45.91.116.5:4298 , piece_id=APSASJC3YKMBJPSARS46GHHWAUJHTTD2ZDF2H3I3SRUYBPKDEM7A, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.793691439 (514h27m45.664774474s) [noise=true , address=77.90.31.12:4590 , piece_id=YYXJHKL477P5I6JMG5R5JINNKL2JSAFMEKRZSXXIJJJUK2GYYQJQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.794234283 (514h27m45.665317318s) [noise=true , address=145.79.227.34:4907 , piece_id=5SHOOI6O2G4AJBWIBCCB4QUJPAURTLLAJ36BBAG3DNLHMTPALUGA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.794782706 (514h27m45.665865741s) [noise=true , address=145.79.227.8:4330 , piece_id=5WLQ7PLSMOPR675V76QX2OM625KRPEPXBLW5X2CH6LOUDAHPMIIA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.795246089 (514h27m45.666329134s) [noise=true , address=45.91.116.10:4083 , piece_id=ZHR2SC4QGEV42TJEGHD2G3D67XNTJWPAXM56BA5YBEWTTOZEW4FA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.795820022 (514h27m45.666903057s) [noise=true , address=45.91.116.13:4155 , piece_id=G6HIMFWVVUD75IQTW7UK7R5C3EPOQ2EKTEZAFPWIRQKLXKLEEERQ, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.796231235 (514h27m45.66731428s) [noise=true , address=45.91.116.6:4975 , piece_id=SFDZ5ATRSW4MWAARFOTJD3APEAIY3MNJIVKHQBZA3A4KUFCGNKQA, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:45.797063530 (514h27m45.668146575s) [noise=true , address=145.79.227.48:4302 , piece_id=IWD2K2BTLHQNBBNPZTUGD3F3UZUWC7ETE3DEQNJM7FGWALRXFW4Q, seg_pos={0 0}, upload=451446]: Uploading piece... Dec 27 01:56:47.518993243 (514h27m47.390076288s) [upload=451446]: Next returned nil segment Dec 27 01:56:47.519009363 (514h27m47.390092398s) [upload=451446]: Waiting for error group managing segments... Dec 27 01:56:47.797138500 (514h27m47.668221545s) [noise=true , address=145.79.227.6:4259 , piece_id=AC5TBXJGL6QKBJPYJWN23ANO4OQNCNZSVLY3PEGC5AE3R57FCZXA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.801346333 (514h27m47.672429368s) [noise=true , address=145.79.227.49:4339 , piece_id=B4V37DEQHYZ6FIK7FLN3LWNP6444GXFJ3XB7QMUJCVQEFC3VRNWQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.808189474 (514h27m47.679272519s) [noise=true , address=145.79.227.48:4302 , piece_id=IWD2K2BTLHQNBBNPZTUGD3F3UZUWC7ETE3DEQNJM7FGWALRXFW4Q, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.813081549 (514h27m47.684164594s) [noise=true , address=145.79.227.8:4330 , piece_id=5WLQ7PLSMOPR675V76QX2OM625KRPEPXBLW5X2CH6LOUDAHPMIIA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.819512034 (514h27m47.690595069s) [noise=true , address=145.79.227.37:4566 , piece_id=VOY4BMWP7PCPPWOVT6WCVDCGATXHPVNN6VYSHHXPAY5LR23WHLIQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.822003978 (514h27m47.693087013s) [noise=true , address=145.79.227.16:4909 , piece_id=QAEBH4FSG2TCSHDSPQSRBQ5LEM3KLLAJYVKOADLYHECLUXL2S3UQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.854846145 (514h27m47.72592918s) [noise=true , address=145.79.227.2:4115 , piece_id=TK7BAFOAVZ5RYH5ICFOYRO4OMMOFSJFDHBI7WEPISV6ZR77KDB4A, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.978276955 (514h27m47.84935999s) [noise=true , address=77.90.31.11:4556 , piece_id=P2OHQJV53WW6NPAQK6KY67U4UASIQYGFJERMAUP5HJHJHXUB34QA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:47.995286223 (514h27m47.866369258s) [noise=true , address=77.90.31.12:4590 , piece_id=YYXJHKL477P5I6JMG5R5JINNKL2JSAFMEKRZSXXIJJJUK2GYYQJQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:48.049358534 (514h27m47.920441569s) [noise=true , address=45.91.116.10:4083 , piece_id=ZHR2SC4QGEV42TJEGHD2G3D67XNTJWPAXM56BA5YBEWTTOZEW4FA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:48.083027645 (514h27m47.95411068s) [noise=true , address=77.90.31.17:4773 , piece_id=BWRG6C4SAA7VX2YDLUWUHIHUS6FF32BERP3GVQAKRSB56G7OKUCQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:48.091653018 (514h27m47.962736083s) [noise=true , address=77.90.31.14:4664 , piece_id=FVSJPKN6SSW35QILLBUUZNKC7RTRVEPBSCQJ6VDTJWNIRFLPOOQA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:48.132338830 (514h27m48.003421875s) [noise=true , address=45.91.116.6:4975 , piece_id=SFDZ5ATRSW4MWAARFOTJD3APEAIY3MNJIVKHQBZA3A4KUFCGNKQA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: <nil> Dec 27 01:56:48.132362034 (514h27m48.003445079s) [seg_pos={0 0}, upload=451446]: Segment reached optimal threshold of 13 pieces. Dec 27 01:56:48.132563818 (514h27m48.003646863s) [noise=true , address=45.91.116.13:4155 , piece_id=G6HIMFWVVUD75IQTW7UK7R5C3EPOQ2EKTEZAFPWIRQKLXKLEEERQ, seg_pos={0 0}, upload=451446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 01:56:48.132572585 (514h27m48.00365563s) [noise=true , address=45.91.116.13:4155 , piece_id=G6HIMFWVVUD75IQTW7UK7R5C3EPOQ2EKTEZAFPWIRQKLXKLEEERQ, seg_pos={0 0}, upload=451446]: Long tail cancellation due to optimal threshold Dec 27 01:56:48.132667806 (514h27m48.003750871s) [noise=true , address=145.79.227.34:4907 , piece_id=5SHOOI6O2G4AJBWIBCCB4QUJPAURTLLAJ36BBAG3DNLHMTPALUGA, seg_pos={0 0}, upload=451446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 01:56:48.132686162 (514h27m48.003769217s) [noise=true , address=145.79.227.34:4907 , piece_id=5SHOOI6O2G4AJBWIBCCB4QUJPAURTLLAJ36BBAG3DNLHMTPALUGA, seg_pos={0 0}, upload=451446]: Long tail cancellation due to optimal threshold Dec 27 01:56:48.132862848 (514h27m48.003945903s) [noise=true , address=45.91.116.5:4298 , piece_id=APSASJC3YKMBJPSARS46GHHWAUJHTTD2ZDF2H3I3SRUYBPKDEM7A, seg_pos={0 0}, upload=451446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:38470->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:38470->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38470->45.91.116.5:4298: use of closed network connection Dec 27 01:56:48.132870472 (514h27m48.003953528s) [noise=true , address=45.91.116.5:4298 , piece_id=APSASJC3YKMBJPSARS46GHHWAUJHTTD2ZDF2H3I3SRUYBPKDEM7A, seg_pos={0 0}, upload=451446]: Long tail cancellation due to optimal threshold Dec 27 01:56:48.132888717 (514h27m48.003971752s) [seg_pos={0 0}, upload=451446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:56:48.132963670 (514h27m48.004046705s) [upload=451446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:56:48.132979229 (514h27m48.004062264s) [upload=451446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:56:48.427704769 (514h27m48.298787804s) [upload=451446]: Done uploading segments... Dec 27 01:56:48.427723114 (514h27m48.298806149s) [upload=451446]: Upload finished. err: <nil> Duration: 30.002305042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X.X] duration: 30.002118161s trace: https://jaeger.dev.storj.io/trace/37f21850fd5443b4 FAIL (2025-12-27 01:56:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:56:15.313252789 (514h27m15.184335824s) [upload=451437]: Starting upload Dec 27 01:56:15.313322793 (514h27m15.184405838s) [upload=451437]: Done starting upload Dec 27 01:56:15.313441589 (514h27m15.184524624s) [upload=451437]: Uploading segments... Dec 27 01:56:15.313553482 (514h27m15.184636527s) [upload=451437]: Got next segment. Inline: false Dec 27 01:56:15.313601634 (514h27m15.184684669s) [upload=451437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:56:15.313607575 (514h27m15.18469061s) [upload=451437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:56:15.560919593 (514h27m15.432002658s) [seg_pos={0 0}, upload=451437]: Begin upload segment... Dec 27 01:56:15.560984968 (514h27m15.432068013s) [seg_pos={0 0}, upload=451437]: Done begin upload segment. Dec 27 01:56:15.561028981 (514h27m15.432112016s) [noise=true , address=145.79.227.5:4222 , piece_id=WFWV3LJJMS5ZM63WZCDCYBKWMYDCYCSZEUI36G5X35APOZ53TXBA, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.561092041 (514h27m15.432175076s) [noise=true , address=145.79.227.45:4231 , piece_id=2BWPZOL3CR6W3EVAGSELM4SW7XGRJKL4V54QMVQYFT3HKAW7D55Q, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.561898497 (514h27m15.432981532s) [noise=true , address=145.79.227.48:4303 , piece_id=T6ZRXVSDWC5MKRI2THRIWBSHVCVKZCPI7B3EWCMET642AT3WPR3Q, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.561939946 (514h27m15.433022981s) [noise=true , address=45.91.116.4:4937 , piece_id=T3ZECHY75XZCUTQOO24IMAEZJEKYHGECFDODUT7MQVAEX2FMES7A, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.562183760 (514h27m15.433266795s) [noise=true , address=77.90.31.10:4520 , piece_id=T7GO4EVXBT2EQIZDNZORJN2GXV2V7GXHI3RMHDUTLLE4KZIMFPOQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.562336420 (514h27m15.433419465s) [noise=true , address=145.79.227.15:4873 , piece_id=FWN27CNTCP3NJZIGCLFUE36UQODEZG23F6T4H75XEC3L66KK2NIQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.562517025 (514h27m15.43360007s) [noise=true , address=77.90.31.17:4773 , piece_id=SDMDWTF2OJUZKIFOIHOIQ7KI2ZGQHL3MLBKFIDP5IWQ2A6AEMMOA, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.562779484 (514h27m15.433862529s) [noise=true , address=45.91.116.13:4153 , piece_id=NLUDNE4KKAH647KWTW24KTUBEVLXYMHMP6H2CL4Z6ZE3KUSB36OA, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.562801296 (514h27m15.433884331s) [noise=true , address=145.79.227.51:4409 , piece_id=BFBORF7WHFAOKF7DV7QQEDP5M6RZ6A6KWD6INTHRXGQSRPO7WLQQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563027126 (514h27m15.434110161s) [noise=true , address=145.79.227.41:4087 , piece_id=FISNVCWK2RXMYKRGRZA7DQHCWXCM77NUPAOIAPYTIMOJ3KNGKZJA, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563233459 (514h27m15.434316504s) [noise=true , address=77.90.31.15:4700 , piece_id=AAUL3OEKLGBKSIEHATZT6L7JGBQV5X4JL5U2UYPS54TMEWE7YC7Q, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563288564 (514h27m15.434371599s) [noise=true , address=145.79.227.43:4158 , piece_id=DPJ4GLXIARCNIFLD462BFQUGJTLRCC6WCSODCY7KE5ZAA62Y37UQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563550101 (514h27m15.434633146s) [noise=true , address=45.91.116.9:4263 , piece_id=4NPSL3VGNRZFCLJK7GQUBSNN32ANLNSAJKCQRJYEZ73MEOAQZ2OQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563647517 (514h27m15.434730562s) [noise=true , address=77.90.31.21:4878 , piece_id=RZOK3A46DATQSNAPYH5SVGV2BDBBNK4O3P27D2YWAE5UXYTEIZJA, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.563798164 (514h27m15.434881199s) [noise=true , address=145.79.227.8:4330 , piece_id=RBQ2ZM5HV67UXL5BLCZCCI4EZHZ6JXIFLEQ2CFLB6TZLRUCV2LFQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:15.564041216 (514h27m15.435124262s) [noise=true , address=145.79.227.42:4122 , piece_id=NU5TP3I7L64GOKMGBQQY6SA3ISHIWET2P7F6MKRD4EAN7FB3BKNQ, seg_pos={0 0}, upload=451437]: Uploading piece... Dec 27 01:56:17.282344084 (514h27m17.153427129s) [upload=451437]: Next returned nil segment Dec 27 01:56:17.282359382 (514h27m17.153442427s) [upload=451437]: Waiting for error group managing segments... Dec 27 01:56:17.563395806 (514h27m17.434478851s) [noise=true , address=145.79.227.45:4231 , piece_id=2BWPZOL3CR6W3EVAGSELM4SW7XGRJKL4V54QMVQYFT3HKAW7D55Q, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.575596438 (514h27m17.446679474s) [noise=true , address=145.79.227.48:4303 , piece_id=T6ZRXVSDWC5MKRI2THRIWBSHVCVKZCPI7B3EWCMET642AT3WPR3Q, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.575768817 (514h27m17.446851852s) [noise=true , address=145.79.227.42:4122 , piece_id=NU5TP3I7L64GOKMGBQQY6SA3ISHIWET2P7F6MKRD4EAN7FB3BKNQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.579645538 (514h27m17.450728583s) [noise=true , address=145.79.227.43:4158 , piece_id=DPJ4GLXIARCNIFLD462BFQUGJTLRCC6WCSODCY7KE5ZAA62Y37UQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.585666614 (514h27m17.456749649s) [noise=true , address=145.79.227.15:4873 , piece_id=FWN27CNTCP3NJZIGCLFUE36UQODEZG23F6T4H75XEC3L66KK2NIQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.586294619 (514h27m17.457377654s) [noise=true , address=145.79.227.8:4330 , piece_id=RBQ2ZM5HV67UXL5BLCZCCI4EZHZ6JXIFLEQ2CFLB6TZLRUCV2LFQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.586610451 (514h27m17.457693496s) [noise=true , address=145.79.227.41:4087 , piece_id=FISNVCWK2RXMYKRGRZA7DQHCWXCM77NUPAOIAPYTIMOJ3KNGKZJA, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.602393684 (514h27m17.473476729s) [noise=true , address=145.79.227.51:4409 , piece_id=BFBORF7WHFAOKF7DV7QQEDP5M6RZ6A6KWD6INTHRXGQSRPO7WLQQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.800316498 (514h27m17.671399533s) [noise=true , address=77.90.31.17:4773 , piece_id=SDMDWTF2OJUZKIFOIHOIQ7KI2ZGQHL3MLBKFIDP5IWQ2A6AEMMOA, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.825826754 (514h27m17.696909789s) [noise=true , address=77.90.31.15:4700 , piece_id=AAUL3OEKLGBKSIEHATZT6L7JGBQV5X4JL5U2UYPS54TMEWE7YC7Q, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.839695722 (514h27m17.710778767s) [noise=true , address=77.90.31.10:4520 , piece_id=T7GO4EVXBT2EQIZDNZORJN2GXV2V7GXHI3RMHDUTLLE4KZIMFPOQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:17.854524889 (514h27m17.725607924s) [noise=true , address=45.91.116.4:4937 , piece_id=T3ZECHY75XZCUTQOO24IMAEZJEKYHGECFDODUT7MQVAEX2FMES7A, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:18.462019416 (514h27m18.333102451s) [noise=true , address=77.90.31.21:4878 , piece_id=RZOK3A46DATQSNAPYH5SVGV2BDBBNK4O3P27D2YWAE5UXYTEIZJA, seg_pos={0 0}, upload=451437]: Done uploading piece. err: <nil> Dec 27 01:56:18.462037932 (514h27m18.333120977s) [seg_pos={0 0}, upload=451437]: Segment reached optimal threshold of 13 pieces. Dec 27 01:56:18.462225649 (514h27m18.333308684s) [noise=true , address=145.79.227.5:4222 , piece_id=WFWV3LJJMS5ZM63WZCDCYBKWMYDCYCSZEUI36G5X35APOZ53TXBA, seg_pos={0 0}, upload=451437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44622->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44622->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44622->145.79.227.5:4222: use of closed network connection Dec 27 01:56:18.462232582 (514h27m18.333315617s) [noise=true , address=145.79.227.5:4222 , piece_id=WFWV3LJJMS5ZM63WZCDCYBKWMYDCYCSZEUI36G5X35APOZ53TXBA, seg_pos={0 0}, upload=451437]: Long tail cancellation due to optimal threshold Dec 27 01:56:18.462330348 (514h27m18.333413383s) [noise=true , address=45.91.116.9:4263 , piece_id=4NPSL3VGNRZFCLJK7GQUBSNN32ANLNSAJKCQRJYEZ73MEOAQZ2OQ, seg_pos={0 0}, upload=451437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 01:56:18.462350547 (514h27m18.333433582s) [noise=true , address=45.91.116.9:4263 , piece_id=4NPSL3VGNRZFCLJK7GQUBSNN32ANLNSAJKCQRJYEZ73MEOAQZ2OQ, seg_pos={0 0}, upload=451437]: Long tail cancellation due to optimal threshold Dec 27 01:56:18.462356618 (514h27m18.333439653s) [noise=true , address=45.91.116.13:4153 , piece_id=NLUDNE4KKAH647KWTW24KTUBEVLXYMHMP6H2CL4Z6ZE3KUSB36OA, seg_pos={0 0}, upload=451437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 01:56:18.462363110 (514h27m18.333446155s) [noise=true , address=45.91.116.13:4153 , piece_id=NLUDNE4KKAH647KWTW24KTUBEVLXYMHMP6H2CL4Z6ZE3KUSB36OA, seg_pos={0 0}, upload=451437]: Long tail cancellation due to optimal threshold Dec 27 01:56:18.462426080 (514h27m18.333509116s) [seg_pos={0 0}, upload=451437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:56:18.462623846 (514h27m18.333706881s) [upload=451437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:56:18.462629879 (514h27m18.333712914s) [upload=451437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:56:18.754779974 (514h27m18.625863019s) [upload=451437]: Done uploading segments... Dec 27 01:56:18.754798549 (514h27m18.625881584s) [upload=451437]: Upload finished. err: <nil> Duration: 30.002177163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X.] duration: 30.00181065s trace: https://jaeger.dev.storj.io/trace/4f4d426bab6e2374 FAIL (2025-12-27 01:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A) context deadline exceeded ======LOGS====== 7 01:54:39.167551092 (514h25m39.038634127s) [upload=451411]: Starting upload Dec 27 01:54:39.167627177 (514h25m39.038710222s) [upload=451411]: Done starting upload Dec 27 01:54:39.167709383 (514h25m39.038792418s) [upload=451411]: Uploading segments... Dec 27 01:54:39.168029993 (514h25m39.039113038s) [upload=451411]: Got next segment. Inline: false Dec 27 01:54:39.168079518 (514h25m39.039162553s) [upload=451411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:54:39.168085900 (514h25m39.039168935s) [upload=451411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:54:39.410131820 (514h25m39.281214875s) [seg_pos={0 0}, upload=451411]: Begin upload segment... Dec 27 01:54:39.410188267 (514h25m39.281271312s) [seg_pos={0 0}, upload=451411]: Done begin upload segment. Dec 27 01:54:39.410199920 (514h25m39.281282955s) [noise=true , address=45.91.116.8:4046 , piece_id=GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.410223905 (514h25m39.28130694s) [noise=true , address=45.91.116.10:4083 , piece_id=ODHQGZ2HTPQM4ZX23PQYLBLN7WTBBOYKE4XN63LQG2I22QE5HHJA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.410717054 (514h25m39.281800089s) [noise=true , address=77.90.31.19:4841 , piece_id=NFNB4VD4PWI5OMWYCZCZ4MN46RG7MXHK5YFQZDZYT5EHPGIHMP7A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.411412859 (514h25m39.282495894s) [noise=true , address=145.79.227.8:4330 , piece_id=LY5LD2BJTTICBYGMJ55PZDQ6MUL36OWWJDFPSXDELHUY5JHGR5SA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.411723019 (514h25m39.282806055s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412029894 (514h25m39.283112939s) [noise=true , address=45.91.116.4:4940 , piece_id=CZLJLAYTLDHXHEAXUSJT6RCOAEY64MBRCXGYMJ3W55CNVIGBMVFQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412328243 (514h25m39.283411278s) [noise=true , address=77.90.31.18:4808 , piece_id=XSVMLC4LLCWQITCB2GDHD5UCLO5XJPLAKOGVQJOXZION53X4DGLA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412696012 (514h25m39.283779047s) [noise=true , address=145.79.227.38:4978 , piece_id=STRJV5EM5PHQVKZ2QFNV3W4GDXSSCCHNBMHTIYATIYP75TG2NNMA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413075595 (514h25m39.28415863s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413424960 (514h25m39.284507995s) [noise=true , address=145.79.227.47:4266 , piece_id=PDIEHZNB4UOM5M4D4VLRIXO5PP4XIH7DEYPSJ3ZHHOTB3BS3OPPQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413936976 (514h25m39.285020021s) [noise=true , address=45.91.116.12:4228 , piece_id=PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414079006 (514h25m39.285162051s) [noise=true , address=145.79.227.11:4731 , piece_id=Y373S366UGFB5LERR4XQZ75C5DJIRA66I6NEUNX4VPXEFMCMU5JQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414579378 (514h25m39.285662413s) [noise=true , address=145.79.227.46:4602 , piece_id=E7ACCG22XEYGG6B7GG3UAEHPKZNO7V7UXOP4EO4R5KJVUGUSHYGA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414689369 (514h25m39.285772404s) [noise=true , address=45.91.116.11:4120 , piece_id=QJD2ODK5AZ4JBS53XAP3WXPCBPZU7DDQL3GEYFV2GPIGEVLAXZPQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414964563 (514h25m39.286047598s) [noise=true , address=145.79.227.16:4909 , piece_id=YH3MHGUBFBXYG7LQ3VCT4OBPPAAM4XPQQH7KT7KVGUEIG77SLEIQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.415242241 (514h25m39.286325276s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:41.123853805 (514h25m40.99493687s) [upload=451411]: Next returned nil segment Dec 27 01:54:41.123872350 (514h25m40.994955415s) [upload=451411]: Waiting for error group managing segments... Dec 27 01:54:41.407701182 (514h25m41.278784227s) [noise=true , address=145.79.227.38:4978 , piece_id=STRJV5EM5PHQVKZ2QFNV3W4GDXSSCCHNBMHTIYATIYP75TG2NNMA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.411287340 (514h25m41.282370375s) [noise=true , address=145.79.227.11:4731 , piece_id=Y373S366UGFB5LERR4XQZ75C5DJIRA66I6NEUNX4VPXEFMCMU5JQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.412494077 (514h25m41.283577112s) [noise=true , address=145.79.227.16:4909 , piece_id=YH3MHGUBFBXYG7LQ3VCT4OBPPAAM4XPQQH7KT7KVGUEIG77SLEIQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.416601629 (514h25m41.287684664s) [noise=true , address=145.79.227.46:4602 , piece_id=E7ACCG22XEYGG6B7GG3UAEHPKZNO7V7UXOP4EO4R5KJVUGUSHYGA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.426766855 (514h25m41.29784989s) [noise=true , address=145.79.227.8:4330 , piece_id=LY5LD2BJTTICBYGMJ55PZDQ6MUL36OWWJDFPSXDELHUY5JHGR5SA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.460604216 (514h25m41.331687251s) [noise=true , address=145.79.227.47:4266 , piece_id=PDIEHZNB4UOM5M4D4VLRIXO5PP4XIH7DEYPSJ3ZHHOTB3BS3OPPQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.638692763 (514h25m41.509775808s) [noise=true , address=45.91.116.12:4228 , piece_id=PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.650855112 (514h25m41.521938167s) [noise=true , address=45.91.116.11:4120 , piece_id=QJD2ODK5AZ4JBS53XAP3WXPCBPZU7DDQL3GEYFV2GPIGEVLAXZPQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.687915910 (514h25m41.558998945s) [noise=true , address=77.90.31.19:4841 , piece_id=NFNB4VD4PWI5OMWYCZCZ4MN46RG7MXHK5YFQZDZYT5EHPGIHMP7A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.744872264 (514h25m41.615955309s) [noise=true , address=45.91.116.4:4940 , piece_id=CZLJLAYTLDHXHEAXUSJT6RCOAEY64MBRCXGYMJ3W55CNVIGBMVFQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.746593391 (514h25m41.617676426s) [noise=true , address=77.90.31.18:4808 , piece_id=XSVMLC4LLCWQITCB2GDHD5UCLO5XJPLAKOGVQJOXZION53X4DGLA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.752203092 (514h25m41.623286127s) [noise=true , address=45.91.116.10:4083 , piece_id=ODHQGZ2HTPQM4ZX23PQYLBLN7WTBBOYKE4XN63LQG2I22QE5HHJA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.759140732 (514h25m41.630223767s) [noise=true , address=45.91.116.8:4046 , piece_id=GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.759160230 (514h25m41.630243275s) [seg_pos={0 0}, upload=451411]: Segment reached optimal threshold of 13 pieces. Dec 27 01:54:41.759361433 (514h25m41.630444478s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection Dec 27 01:54:41.759369278 (514h25m41.630452323s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759438520 (514h25m41.630521555s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 01:54:41.759450923 (514h25m41.630533968s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759591791 (514h25m41.630674826s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 01:54:41.759597884 (514h25m41.630680929s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759635746 (514h25m41.630718811s) [seg_pos={0 0}, upload=451411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:54:41.759743461 (514h25m41.630826496s) [upload=451411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:54:41.759750143 (514h25m41.630833188s) [upload=451411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:54:42.042301651 (514h25m41.913384686s) [upload=451411]: Done uploading segments... Dec 27 01:54:42.042321129 (514h25m41.913404164s) [upload=451411]: Upload finished. err: <nil> Duration: 30.001837501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...X] duration: 30.002707405s trace: https://jaeger.dev.storj.io/trace/4f4d426bab6e2374 FAIL (2025-12-27 01:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A) context deadline exceeded ======LOGS====== 7 01:54:39.167551092 (514h25m39.038634127s) [upload=451411]: Starting upload Dec 27 01:54:39.167627177 (514h25m39.038710222s) [upload=451411]: Done starting upload Dec 27 01:54:39.167709383 (514h25m39.038792418s) [upload=451411]: Uploading segments... Dec 27 01:54:39.168029993 (514h25m39.039113038s) [upload=451411]: Got next segment. Inline: false Dec 27 01:54:39.168079518 (514h25m39.039162553s) [upload=451411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:54:39.168085900 (514h25m39.039168935s) [upload=451411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:54:39.410131820 (514h25m39.281214875s) [seg_pos={0 0}, upload=451411]: Begin upload segment... Dec 27 01:54:39.410188267 (514h25m39.281271312s) [seg_pos={0 0}, upload=451411]: Done begin upload segment. Dec 27 01:54:39.410199920 (514h25m39.281282955s) [noise=true , address=45.91.116.8:4046 , piece_id=GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.410223905 (514h25m39.28130694s) [noise=true , address=45.91.116.10:4083 , piece_id=ODHQGZ2HTPQM4ZX23PQYLBLN7WTBBOYKE4XN63LQG2I22QE5HHJA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.410717054 (514h25m39.281800089s) [noise=true , address=77.90.31.19:4841 , piece_id=NFNB4VD4PWI5OMWYCZCZ4MN46RG7MXHK5YFQZDZYT5EHPGIHMP7A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.411412859 (514h25m39.282495894s) [noise=true , address=145.79.227.8:4330 , piece_id=LY5LD2BJTTICBYGMJ55PZDQ6MUL36OWWJDFPSXDELHUY5JHGR5SA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.411723019 (514h25m39.282806055s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412029894 (514h25m39.283112939s) [noise=true , address=45.91.116.4:4940 , piece_id=CZLJLAYTLDHXHEAXUSJT6RCOAEY64MBRCXGYMJ3W55CNVIGBMVFQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412328243 (514h25m39.283411278s) [noise=true , address=77.90.31.18:4808 , piece_id=XSVMLC4LLCWQITCB2GDHD5UCLO5XJPLAKOGVQJOXZION53X4DGLA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.412696012 (514h25m39.283779047s) [noise=true , address=145.79.227.38:4978 , piece_id=STRJV5EM5PHQVKZ2QFNV3W4GDXSSCCHNBMHTIYATIYP75TG2NNMA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413075595 (514h25m39.28415863s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413424960 (514h25m39.284507995s) [noise=true , address=145.79.227.47:4266 , piece_id=PDIEHZNB4UOM5M4D4VLRIXO5PP4XIH7DEYPSJ3ZHHOTB3BS3OPPQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.413936976 (514h25m39.285020021s) [noise=true , address=45.91.116.12:4228 , piece_id=PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414079006 (514h25m39.285162051s) [noise=true , address=145.79.227.11:4731 , piece_id=Y373S366UGFB5LERR4XQZ75C5DJIRA66I6NEUNX4VPXEFMCMU5JQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414579378 (514h25m39.285662413s) [noise=true , address=145.79.227.46:4602 , piece_id=E7ACCG22XEYGG6B7GG3UAEHPKZNO7V7UXOP4EO4R5KJVUGUSHYGA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414689369 (514h25m39.285772404s) [noise=true , address=45.91.116.11:4120 , piece_id=QJD2ODK5AZ4JBS53XAP3WXPCBPZU7DDQL3GEYFV2GPIGEVLAXZPQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.414964563 (514h25m39.286047598s) [noise=true , address=145.79.227.16:4909 , piece_id=YH3MHGUBFBXYG7LQ3VCT4OBPPAAM4XPQQH7KT7KVGUEIG77SLEIQ, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:39.415242241 (514h25m39.286325276s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Uploading piece... Dec 27 01:54:41.123853805 (514h25m40.99493687s) [upload=451411]: Next returned nil segment Dec 27 01:54:41.123872350 (514h25m40.994955415s) [upload=451411]: Waiting for error group managing segments... Dec 27 01:54:41.407701182 (514h25m41.278784227s) [noise=true , address=145.79.227.38:4978 , piece_id=STRJV5EM5PHQVKZ2QFNV3W4GDXSSCCHNBMHTIYATIYP75TG2NNMA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.411287340 (514h25m41.282370375s) [noise=true , address=145.79.227.11:4731 , piece_id=Y373S366UGFB5LERR4XQZ75C5DJIRA66I6NEUNX4VPXEFMCMU5JQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.412494077 (514h25m41.283577112s) [noise=true , address=145.79.227.16:4909 , piece_id=YH3MHGUBFBXYG7LQ3VCT4OBPPAAM4XPQQH7KT7KVGUEIG77SLEIQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.416601629 (514h25m41.287684664s) [noise=true , address=145.79.227.46:4602 , piece_id=E7ACCG22XEYGG6B7GG3UAEHPKZNO7V7UXOP4EO4R5KJVUGUSHYGA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.426766855 (514h25m41.29784989s) [noise=true , address=145.79.227.8:4330 , piece_id=LY5LD2BJTTICBYGMJ55PZDQ6MUL36OWWJDFPSXDELHUY5JHGR5SA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.460604216 (514h25m41.331687251s) [noise=true , address=145.79.227.47:4266 , piece_id=PDIEHZNB4UOM5M4D4VLRIXO5PP4XIH7DEYPSJ3ZHHOTB3BS3OPPQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.638692763 (514h25m41.509775808s) [noise=true , address=45.91.116.12:4228 , piece_id=PXCJIY2UQLOUNJXGGJ5SJW7JFPVJE7FZCH2TW37PZL6RQCKWVB6A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.650855112 (514h25m41.521938167s) [noise=true , address=45.91.116.11:4120 , piece_id=QJD2ODK5AZ4JBS53XAP3WXPCBPZU7DDQL3GEYFV2GPIGEVLAXZPQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.687915910 (514h25m41.558998945s) [noise=true , address=77.90.31.19:4841 , piece_id=NFNB4VD4PWI5OMWYCZCZ4MN46RG7MXHK5YFQZDZYT5EHPGIHMP7A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.744872264 (514h25m41.615955309s) [noise=true , address=45.91.116.4:4940 , piece_id=CZLJLAYTLDHXHEAXUSJT6RCOAEY64MBRCXGYMJ3W55CNVIGBMVFQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.746593391 (514h25m41.617676426s) [noise=true , address=77.90.31.18:4808 , piece_id=XSVMLC4LLCWQITCB2GDHD5UCLO5XJPLAKOGVQJOXZION53X4DGLA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.752203092 (514h25m41.623286127s) [noise=true , address=45.91.116.10:4083 , piece_id=ODHQGZ2HTPQM4ZX23PQYLBLN7WTBBOYKE4XN63LQG2I22QE5HHJA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.759140732 (514h25m41.630223767s) [noise=true , address=45.91.116.8:4046 , piece_id=GSU4AA6XXUTSYVBTJL4QJFZZIEVUOISNER77DROYBNQAGHUGUI2A, seg_pos={0 0}, upload=451411]: Done uploading piece. err: <nil> Dec 27 01:54:41.759160230 (514h25m41.630243275s) [seg_pos={0 0}, upload=451411]: Segment reached optimal threshold of 13 pieces. Dec 27 01:54:41.759361433 (514h25m41.630444478s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46888->45.91.116.14:4190: use of closed network connection Dec 27 01:54:41.759369278 (514h25m41.630452323s) [noise=true , address=45.91.116.14:4190 , piece_id=XN7QORBB2IPBG7BJBCRCWBI7G2F7TTESELE2JXLL3E7QN6AVUWBQ, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759438520 (514h25m41.630521555s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 01:54:41.759450923 (514h25m41.630533968s) [noise=true , address=77.90.31.10:4520 , piece_id=MAC7BMNXK7MMHZ4R75TONYS4ARP4VNP6DLT5NR3KRJTF3S5NM6TA, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759591791 (514h25m41.630674826s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 01:54:41.759597884 (514h25m41.630680929s) [noise=true , address=45.91.116.13:4156 , piece_id=4MKNC7DE7MHHOMHH7PIQQIPQOKIJH72Y674PJKK3GI3GUJCQGK7Q, seg_pos={0 0}, upload=451411]: Long tail cancellation due to optimal threshold Dec 27 01:54:41.759635746 (514h25m41.630718811s) [seg_pos={0 0}, upload=451411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:54:41.759743461 (514h25m41.630826496s) [upload=451411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:54:41.759750143 (514h25m41.630833188s) [upload=451411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:54:42.042301651 (514h25m41.913384686s) [upload=451411]: Done uploading segments... Dec 27 01:54:42.042321129 (514h25m41.913404164s) [upload=451411]: Upload finished. err: <nil> Duration: 30.002797437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X...] duration: 7.569097376s trace: https://jaeger.dev.storj.io/trace/7e5db941ad190eb6 FAIL (2025-12-27 01:54:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ) context deadline exceeded ======LOGS====== 514h23m0.279043974s) [upload=451372]: Uploading segments... Dec 27 01:52:00.408120864 (514h23m0.279203899s) [upload=451372]: Got next segment. Inline: false Dec 27 01:52:00.408154609 (514h23m0.279237654s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:52:00.408159718 (514h23m0.279242763s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:52:00.657079687 (514h23m0.528162742s) [seg_pos={0 0}, upload=451372]: Begin upload segment... Dec 27 01:52:00.657152305 (514h23m0.52823536s) [seg_pos={0 0}, upload=451372]: Done begin upload segment. Dec 27 01:52:00.657240063 (514h23m0.528323098s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657277083 (514h23m0.528360128s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657923544 (514h23m0.529006579s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658034536 (514h23m0.529117581s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658317384 (514h23m0.529400429s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658493600 (514h23m0.529576645s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658812737 (514h23m0.529895782s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659016826 (514h23m0.530099861s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659238959 (514h23m0.530322004s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659494465 (514h23m0.53057751s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659641816 (514h23m0.530724861s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659871673 (514h23m0.530954718s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660068649 (514h23m0.531151694s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660384039 (514h23m0.531467074s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660477958 (514h23m0.531560993s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660802396 (514h23m0.531885451s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:02.373619547 (514h23m2.244702592s) [upload=451372]: Next returned nil segment Dec 27 01:52:02.373637011 (514h23m2.244720046s) [upload=451372]: Waiting for error group managing segments... Dec 27 01:52:02.659779737 (514h23m2.530862772s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.660283156 (514h23m2.531366201s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.661512647 (514h23m2.532595692s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.664667615 (514h23m2.53575065s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.667025764 (514h23m2.538108809s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.675656399 (514h23m2.546739444s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.693665590 (514h23m2.564748635s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.858370071 (514h23m2.729453116s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.899553541 (514h23m2.770636576s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.913255501 (514h23m2.784338546s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.952048649 (514h23m2.823131694s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.960496825 (514h23m2.83157987s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985936767 (514h23m2.857019812s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985956013 (514h23m2.857039058s) [seg_pos={0 0}, upload=451372]: Segment reached optimal threshold of 13 pieces. Dec 27 01:52:02.986222310 (514h23m2.857305345s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 01:52:02.986230335 (514h23m2.85731337s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986268307 (514h23m2.857351353s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection Dec 27 01:52:02.986284379 (514h23m2.857367424s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986614457 (514h23m2.857697502s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection Dec 27 01:52:02.986623043 (514h23m2.857706078s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986648061 (514h23m2.857731096s) [seg_pos={0 0}, upload=451372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:52:02.986730137 (514h23m2.857813182s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:52:02.986735416 (514h23m2.857818462s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:52:03.278686033 (514h23m3.149769088s) [upload=451372]: Done uploading segments... Dec 27 01:52:03.278720689 (514h23m3.149803774s) [upload=451372]: Upload finished. err: <nil> Duration: 7.569121352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X..] duration: 5.982906977s trace: https://jaeger.dev.storj.io/trace/7e5db941ad190eb6 FAIL (2025-12-27 01:53:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ) context deadline exceeded ======LOGS====== 514h23m0.279043974s) [upload=451372]: Uploading segments... Dec 27 01:52:00.408120864 (514h23m0.279203899s) [upload=451372]: Got next segment. Inline: false Dec 27 01:52:00.408154609 (514h23m0.279237654s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:52:00.408159718 (514h23m0.279242763s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:52:00.657079687 (514h23m0.528162742s) [seg_pos={0 0}, upload=451372]: Begin upload segment... Dec 27 01:52:00.657152305 (514h23m0.52823536s) [seg_pos={0 0}, upload=451372]: Done begin upload segment. Dec 27 01:52:00.657240063 (514h23m0.528323098s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657277083 (514h23m0.528360128s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657923544 (514h23m0.529006579s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658034536 (514h23m0.529117581s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658317384 (514h23m0.529400429s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658493600 (514h23m0.529576645s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658812737 (514h23m0.529895782s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659016826 (514h23m0.530099861s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659238959 (514h23m0.530322004s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659494465 (514h23m0.53057751s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659641816 (514h23m0.530724861s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659871673 (514h23m0.530954718s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660068649 (514h23m0.531151694s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660384039 (514h23m0.531467074s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660477958 (514h23m0.531560993s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660802396 (514h23m0.531885451s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:02.373619547 (514h23m2.244702592s) [upload=451372]: Next returned nil segment Dec 27 01:52:02.373637011 (514h23m2.244720046s) [upload=451372]: Waiting for error group managing segments... Dec 27 01:52:02.659779737 (514h23m2.530862772s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.660283156 (514h23m2.531366201s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.661512647 (514h23m2.532595692s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.664667615 (514h23m2.53575065s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.667025764 (514h23m2.538108809s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.675656399 (514h23m2.546739444s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.693665590 (514h23m2.564748635s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.858370071 (514h23m2.729453116s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.899553541 (514h23m2.770636576s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.913255501 (514h23m2.784338546s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.952048649 (514h23m2.823131694s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.960496825 (514h23m2.83157987s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985936767 (514h23m2.857019812s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985956013 (514h23m2.857039058s) [seg_pos={0 0}, upload=451372]: Segment reached optimal threshold of 13 pieces. Dec 27 01:52:02.986222310 (514h23m2.857305345s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 01:52:02.986230335 (514h23m2.85731337s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986268307 (514h23m2.857351353s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection Dec 27 01:52:02.986284379 (514h23m2.857367424s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986614457 (514h23m2.857697502s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection Dec 27 01:52:02.986623043 (514h23m2.857706078s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986648061 (514h23m2.857731096s) [seg_pos={0 0}, upload=451372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:52:02.986730137 (514h23m2.857813182s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:52:02.986735416 (514h23m2.857818462s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:52:03.278686033 (514h23m3.149769088s) [upload=451372]: Done uploading segments... Dec 27 01:52:03.278720689 (514h23m3.149803774s) [upload=451372]: Upload finished. err: <nil> Duration: 5.982926613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X.] duration: 5.781807294s trace: https://jaeger.dev.storj.io/trace/7e5db941ad190eb6 FAIL (2025-12-27 01:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ) context deadline exceeded ======LOGS====== 514h23m0.279043974s) [upload=451372]: Uploading segments... Dec 27 01:52:00.408120864 (514h23m0.279203899s) [upload=451372]: Got next segment. Inline: false Dec 27 01:52:00.408154609 (514h23m0.279237654s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:52:00.408159718 (514h23m0.279242763s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:52:00.657079687 (514h23m0.528162742s) [seg_pos={0 0}, upload=451372]: Begin upload segment... Dec 27 01:52:00.657152305 (514h23m0.52823536s) [seg_pos={0 0}, upload=451372]: Done begin upload segment. Dec 27 01:52:00.657240063 (514h23m0.528323098s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657277083 (514h23m0.528360128s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657923544 (514h23m0.529006579s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658034536 (514h23m0.529117581s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658317384 (514h23m0.529400429s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658493600 (514h23m0.529576645s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658812737 (514h23m0.529895782s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659016826 (514h23m0.530099861s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659238959 (514h23m0.530322004s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659494465 (514h23m0.53057751s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659641816 (514h23m0.530724861s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659871673 (514h23m0.530954718s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660068649 (514h23m0.531151694s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660384039 (514h23m0.531467074s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660477958 (514h23m0.531560993s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660802396 (514h23m0.531885451s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:02.373619547 (514h23m2.244702592s) [upload=451372]: Next returned nil segment Dec 27 01:52:02.373637011 (514h23m2.244720046s) [upload=451372]: Waiting for error group managing segments... Dec 27 01:52:02.659779737 (514h23m2.530862772s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.660283156 (514h23m2.531366201s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.661512647 (514h23m2.532595692s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.664667615 (514h23m2.53575065s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.667025764 (514h23m2.538108809s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.675656399 (514h23m2.546739444s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.693665590 (514h23m2.564748635s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.858370071 (514h23m2.729453116s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.899553541 (514h23m2.770636576s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.913255501 (514h23m2.784338546s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.952048649 (514h23m2.823131694s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.960496825 (514h23m2.83157987s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985936767 (514h23m2.857019812s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985956013 (514h23m2.857039058s) [seg_pos={0 0}, upload=451372]: Segment reached optimal threshold of 13 pieces. Dec 27 01:52:02.986222310 (514h23m2.857305345s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 01:52:02.986230335 (514h23m2.85731337s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986268307 (514h23m2.857351353s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection Dec 27 01:52:02.986284379 (514h23m2.857367424s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986614457 (514h23m2.857697502s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection Dec 27 01:52:02.986623043 (514h23m2.857706078s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986648061 (514h23m2.857731096s) [seg_pos={0 0}, upload=451372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:52:02.986730137 (514h23m2.857813182s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:52:02.986735416 (514h23m2.857818462s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:52:03.278686033 (514h23m3.149769088s) [upload=451372]: Done uploading segments... Dec 27 01:52:03.278720689 (514h23m3.149803774s) [upload=451372]: Upload finished. err: <nil> Duration: 5.781839676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...X] duration: 30.004408907s trace: https://jaeger.dev.storj.io/trace/7e5db941ad190eb6 FAIL (2025-12-27 01:52:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ) context deadline exceeded ======LOGS====== 514h23m0.279043974s) [upload=451372]: Uploading segments... Dec 27 01:52:00.408120864 (514h23m0.279203899s) [upload=451372]: Got next segment. Inline: false Dec 27 01:52:00.408154609 (514h23m0.279237654s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:52:00.408159718 (514h23m0.279242763s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:52:00.657079687 (514h23m0.528162742s) [seg_pos={0 0}, upload=451372]: Begin upload segment... Dec 27 01:52:00.657152305 (514h23m0.52823536s) [seg_pos={0 0}, upload=451372]: Done begin upload segment. Dec 27 01:52:00.657240063 (514h23m0.528323098s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657277083 (514h23m0.528360128s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.657923544 (514h23m0.529006579s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658034536 (514h23m0.529117581s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658317384 (514h23m0.529400429s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658493600 (514h23m0.529576645s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.658812737 (514h23m0.529895782s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659016826 (514h23m0.530099861s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659238959 (514h23m0.530322004s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659494465 (514h23m0.53057751s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659641816 (514h23m0.530724861s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.659871673 (514h23m0.530954718s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660068649 (514h23m0.531151694s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660384039 (514h23m0.531467074s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660477958 (514h23m0.531560993s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:00.660802396 (514h23m0.531885451s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Uploading piece... Dec 27 01:52:02.373619547 (514h23m2.244702592s) [upload=451372]: Next returned nil segment Dec 27 01:52:02.373637011 (514h23m2.244720046s) [upload=451372]: Waiting for error group managing segments... Dec 27 01:52:02.659779737 (514h23m2.530862772s) [noise=true , address=145.79.227.8:4330 , piece_id=MRFVYPXOLRSGK6EPVUAYL4RDXENUQVMYWCBJ7RY2E56BGKI23TDQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.660283156 (514h23m2.531366201s) [noise=true , address=145.79.227.48:4302 , piece_id=KOJNR52OCDHPEDEHZKWVV5CWFBB2SMMIRMYE4WCSVC4ACKYUNKRA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.661512647 (514h23m2.532595692s) [noise=true , address=145.79.227.38:4977 , piece_id=CEZTHBYY5VDHARQSUI5FE4GXMDIL7CEFQ6GQ5TO5XAT7RFBAGY4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.664667615 (514h23m2.53575065s) [noise=true , address=145.79.227.46:4601 , piece_id=FVAZY4Z2B3DLWIGVW2CP5PW4EHQGLPGAAPVBDCF6ZDRZFMHV427A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.667025764 (514h23m2.538108809s) [noise=true , address=145.79.227.15:4874 , piece_id=YHGXI5F7OXG2DUTXRPOAMZWU5GJOVGYKBOIO5W2GQZA42RLLU44A, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.675656399 (514h23m2.546739444s) [noise=true , address=145.79.227.39:4013 , piece_id=H6WRRLUTV5NTB64BVL7NL7EHEOR46P4CRXHLWMEZ4HFP53CTPZZQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.693665590 (514h23m2.564748635s) [noise=true , address=145.79.227.41:4085 , piece_id=DEHZOVVLV2CVQWS46RPDZXAGZAC4T6PSQGOAR7UKINQVEVWSTX4Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.858370071 (514h23m2.729453116s) [noise=true , address=77.90.31.11:4556 , piece_id=RFBVHHGHBPP4GSDU7GRLHPR45BWEZUOZFZTNQWEWIBEDT2YC5FRQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.899553541 (514h23m2.770636576s) [noise=true , address=77.90.31.10:4517 , piece_id=JZEZDC7BMZKUMFFEHMXA4DRNDOWGS6TCXLONLSFOHQ5NBZHVCOVQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.913255501 (514h23m2.784338546s) [noise=true , address=45.91.116.14:4192 , piece_id=C7LH2QJQHXEMLGOD3AVFZYOJJERGS66NX7OUAITHWTQBAFDGLN2Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.952048649 (514h23m2.823131694s) [noise=true , address=45.91.116.11:4119 , piece_id=D3T5FGAIQU7GVSC272DFYSONLVUZF2EYWU6HAMKOZ6EMV63TP5NQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.960496825 (514h23m2.83157987s) [noise=true , address=77.90.31.19:4842 , piece_id=WZ2AYC3BDQQL36WHEFB447T7RW4N2L6D5B2QUWO6BAASQNUGVD5Q, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985936767 (514h23m2.857019812s) [noise=true , address=45.91.116.10:4083 , piece_id=5OXHMVSCAIUBE47WEJNJXO4CSCWQ2BEBSZP5HYB4GUOUUSGMJPZA, seg_pos={0 0}, upload=451372]: Done uploading piece. err: <nil> Dec 27 01:52:02.985956013 (514h23m2.857039058s) [seg_pos={0 0}, upload=451372]: Segment reached optimal threshold of 13 pieces. Dec 27 01:52:02.986222310 (514h23m2.857305345s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 01:52:02.986230335 (514h23m2.85731337s) [noise=true , address=145.79.227.43:4158 , piece_id=TJFNMBNR36ZFFZVKMKS2VBUPKEY4P3AQJ2VGSLD2SBRF6THFQXQQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986268307 (514h23m2.857351353s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50510->145.79.227.4:4185: use of closed network connection Dec 27 01:52:02.986284379 (514h23m2.857367424s) [noise=true , address=145.79.227.4:4185 , piece_id=3YXGCQYOMEV55KID766PGDZATQKQAWY23QHWNQJBT3ZEVSW4NZAQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986614457 (514h23m2.857697502s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39418->77.90.31.14:4663: use of closed network connection Dec 27 01:52:02.986623043 (514h23m2.857706078s) [noise=true , address=77.90.31.14:4663 , piece_id=NAFPZDEFZXISR3NGBMKOB2XRQZFKHNIB6PAEAY6JPVYQK25OZBJQ, seg_pos={0 0}, upload=451372]: Long tail cancellation due to optimal threshold Dec 27 01:52:02.986648061 (514h23m2.857731096s) [seg_pos={0 0}, upload=451372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:52:02.986730137 (514h23m2.857813182s) [upload=451372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:52:02.986735416 (514h23m2.857818462s) [upload=451372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:52:03.278686033 (514h23m3.149769088s) [upload=451372]: Done uploading segments... Dec 27 01:52:03.278720689 (514h23m3.149803774s) [upload=451372]: Upload finished. err: <nil> Duration: 30.004431299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX...] duration: 6.126578711s trace: https://jaeger.dev.storj.io/trace/6ea0b5f84a9a0fad FAIL (2025-12-27 01:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 01:49:22.764976219 (514h20m22.636059254s) [upload=451331]: Starting upload Dec 27 01:49:22.765108361 (514h20m22.636191396s) [upload=451331]: Done starting upload Dec 27 01:49:22.765378355 (514h20m22.63646139s) [upload=451331]: Uploading segments... Dec 27 01:49:22.765450823 (514h20m22.636533878s) [upload=451331]: Got next segment. Inline: false Dec 27 01:49:22.765478085 (514h20m22.63656112s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:49:22.765482784 (514h20m22.636565829s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:49:23.013284613 (514h20m22.884367648s) [seg_pos={0 0}, upload=451331]: Begin upload segment... Dec 27 01:49:23.013338385 (514h20m22.88442143s) [seg_pos={0 0}, upload=451331]: Done begin upload segment. Dec 27 01:49:23.013369725 (514h20m22.88445277s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013382700 (514h20m22.884465735s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013596747 (514h20m22.884679792s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014036915 (514h20m22.88511995s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014774921 (514h20m22.885857956s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015178709 (514h20m22.886261754s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015671768 (514h20m22.886754803s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016104583 (514h20m22.887187618s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016573124 (514h20m22.887656169s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.017594740 (514h20m22.888677775s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.018155288 (514h20m22.889238333s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.019076802 (514h20m22.890159847s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020007875 (514h20m22.89109093s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020871730 (514h20m22.891954775s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.021955734 (514h20m22.893038779s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.023529220 (514h20m22.894612265s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:24.646694782 (514h20m24.517777817s) [upload=451331]: Next returned nil segment Dec 27 01:49:24.646715311 (514h20m24.517798346s) [upload=451331]: Waiting for error group managing segments... Dec 27 01:49:25.026519311 (514h20m24.897602356s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032030675 (514h20m24.90311372s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032385070 (514h20m24.903468105s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.034087902 (514h20m24.905170947s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.036163545 (514h20m24.90724659s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.037637121 (514h20m24.908720166s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.042262888 (514h20m24.913345923s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.055528678 (514h20m24.926611723s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.199505079 (514h20m25.070588124s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.208923554 (514h20m25.080006589s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.250704229 (514h20m25.121787265s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.251548908 (514h20m25.122631943s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304930404 (514h20m25.176013439s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304949670 (514h20m25.176032705s) [seg_pos={0 0}, upload=451331]: Segment reached optimal threshold of 13 pieces. Dec 27 01:49:25.305125835 (514h20m25.17620887s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection Dec 27 01:49:25.305132128 (514h20m25.176215163s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305398585 (514h20m25.17648162s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305409256 (514h20m25.176492291s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305430736 (514h20m25.176513771s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305439202 (514h20m25.176522257s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305457618 (514h20m25.176540653s) [seg_pos={0 0}, upload=451331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:49:25.305543511 (514h20m25.176626556s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:49:25.305550996 (514h20m25.176634031s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:49:25.605940909 (514h20m25.477023954s) [upload=451331]: Done uploading segments... Dec 27 01:49:25.605963783 (514h20m25.477046838s) [upload=451331]: Upload finished. err: <nil> Duration: 6.126606896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX..] duration: 6.771602505s trace: https://jaeger.dev.storj.io/trace/6ea0b5f84a9a0fad FAIL (2025-12-27 01:50:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 01:49:22.764976219 (514h20m22.636059254s) [upload=451331]: Starting upload Dec 27 01:49:22.765108361 (514h20m22.636191396s) [upload=451331]: Done starting upload Dec 27 01:49:22.765378355 (514h20m22.63646139s) [upload=451331]: Uploading segments... Dec 27 01:49:22.765450823 (514h20m22.636533878s) [upload=451331]: Got next segment. Inline: false Dec 27 01:49:22.765478085 (514h20m22.63656112s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:49:22.765482784 (514h20m22.636565829s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:49:23.013284613 (514h20m22.884367648s) [seg_pos={0 0}, upload=451331]: Begin upload segment... Dec 27 01:49:23.013338385 (514h20m22.88442143s) [seg_pos={0 0}, upload=451331]: Done begin upload segment. Dec 27 01:49:23.013369725 (514h20m22.88445277s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013382700 (514h20m22.884465735s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013596747 (514h20m22.884679792s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014036915 (514h20m22.88511995s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014774921 (514h20m22.885857956s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015178709 (514h20m22.886261754s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015671768 (514h20m22.886754803s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016104583 (514h20m22.887187618s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016573124 (514h20m22.887656169s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.017594740 (514h20m22.888677775s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.018155288 (514h20m22.889238333s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.019076802 (514h20m22.890159847s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020007875 (514h20m22.89109093s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020871730 (514h20m22.891954775s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.021955734 (514h20m22.893038779s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.023529220 (514h20m22.894612265s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:24.646694782 (514h20m24.517777817s) [upload=451331]: Next returned nil segment Dec 27 01:49:24.646715311 (514h20m24.517798346s) [upload=451331]: Waiting for error group managing segments... Dec 27 01:49:25.026519311 (514h20m24.897602356s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032030675 (514h20m24.90311372s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032385070 (514h20m24.903468105s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.034087902 (514h20m24.905170947s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.036163545 (514h20m24.90724659s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.037637121 (514h20m24.908720166s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.042262888 (514h20m24.913345923s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.055528678 (514h20m24.926611723s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.199505079 (514h20m25.070588124s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.208923554 (514h20m25.080006589s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.250704229 (514h20m25.121787265s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.251548908 (514h20m25.122631943s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304930404 (514h20m25.176013439s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304949670 (514h20m25.176032705s) [seg_pos={0 0}, upload=451331]: Segment reached optimal threshold of 13 pieces. Dec 27 01:49:25.305125835 (514h20m25.17620887s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection Dec 27 01:49:25.305132128 (514h20m25.176215163s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305398585 (514h20m25.17648162s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305409256 (514h20m25.176492291s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305430736 (514h20m25.176513771s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305439202 (514h20m25.176522257s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305457618 (514h20m25.176540653s) [seg_pos={0 0}, upload=451331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:49:25.305543511 (514h20m25.176626556s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:49:25.305550996 (514h20m25.176634031s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:49:25.605940909 (514h20m25.477023954s) [upload=451331]: Done uploading segments... Dec 27 01:49:25.605963783 (514h20m25.477046838s) [upload=451331]: Upload finished. err: <nil> Duration: 6.771629546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX.] duration: 6.580363926s trace: https://jaeger.dev.storj.io/trace/6ea0b5f84a9a0fad FAIL (2025-12-27 01:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 01:49:22.764976219 (514h20m22.636059254s) [upload=451331]: Starting upload Dec 27 01:49:22.765108361 (514h20m22.636191396s) [upload=451331]: Done starting upload Dec 27 01:49:22.765378355 (514h20m22.63646139s) [upload=451331]: Uploading segments... Dec 27 01:49:22.765450823 (514h20m22.636533878s) [upload=451331]: Got next segment. Inline: false Dec 27 01:49:22.765478085 (514h20m22.63656112s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:49:22.765482784 (514h20m22.636565829s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:49:23.013284613 (514h20m22.884367648s) [seg_pos={0 0}, upload=451331]: Begin upload segment... Dec 27 01:49:23.013338385 (514h20m22.88442143s) [seg_pos={0 0}, upload=451331]: Done begin upload segment. Dec 27 01:49:23.013369725 (514h20m22.88445277s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013382700 (514h20m22.884465735s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013596747 (514h20m22.884679792s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014036915 (514h20m22.88511995s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014774921 (514h20m22.885857956s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015178709 (514h20m22.886261754s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015671768 (514h20m22.886754803s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016104583 (514h20m22.887187618s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016573124 (514h20m22.887656169s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.017594740 (514h20m22.888677775s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.018155288 (514h20m22.889238333s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.019076802 (514h20m22.890159847s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020007875 (514h20m22.89109093s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020871730 (514h20m22.891954775s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.021955734 (514h20m22.893038779s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.023529220 (514h20m22.894612265s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:24.646694782 (514h20m24.517777817s) [upload=451331]: Next returned nil segment Dec 27 01:49:24.646715311 (514h20m24.517798346s) [upload=451331]: Waiting for error group managing segments... Dec 27 01:49:25.026519311 (514h20m24.897602356s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032030675 (514h20m24.90311372s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032385070 (514h20m24.903468105s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.034087902 (514h20m24.905170947s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.036163545 (514h20m24.90724659s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.037637121 (514h20m24.908720166s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.042262888 (514h20m24.913345923s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.055528678 (514h20m24.926611723s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.199505079 (514h20m25.070588124s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.208923554 (514h20m25.080006589s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.250704229 (514h20m25.121787265s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.251548908 (514h20m25.122631943s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304930404 (514h20m25.176013439s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304949670 (514h20m25.176032705s) [seg_pos={0 0}, upload=451331]: Segment reached optimal threshold of 13 pieces. Dec 27 01:49:25.305125835 (514h20m25.17620887s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection Dec 27 01:49:25.305132128 (514h20m25.176215163s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305398585 (514h20m25.17648162s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305409256 (514h20m25.176492291s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305430736 (514h20m25.176513771s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305439202 (514h20m25.176522257s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305457618 (514h20m25.176540653s) [seg_pos={0 0}, upload=451331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:49:25.305543511 (514h20m25.176626556s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:49:25.305550996 (514h20m25.176634031s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:49:25.605940909 (514h20m25.477023954s) [upload=451331]: Done uploading segments... Dec 27 01:49:25.605963783 (514h20m25.477046838s) [upload=451331]: Upload finished. err: <nil> Duration: 6.580390888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...XX] duration: 30.003400647s trace: https://jaeger.dev.storj.io/trace/6ea0b5f84a9a0fad FAIL (2025-12-27 01:49:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 27 01:49:22.764976219 (514h20m22.636059254s) [upload=451331]: Starting upload Dec 27 01:49:22.765108361 (514h20m22.636191396s) [upload=451331]: Done starting upload Dec 27 01:49:22.765378355 (514h20m22.63646139s) [upload=451331]: Uploading segments... Dec 27 01:49:22.765450823 (514h20m22.636533878s) [upload=451331]: Got next segment. Inline: false Dec 27 01:49:22.765478085 (514h20m22.63656112s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:49:22.765482784 (514h20m22.636565829s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:49:23.013284613 (514h20m22.884367648s) [seg_pos={0 0}, upload=451331]: Begin upload segment... Dec 27 01:49:23.013338385 (514h20m22.88442143s) [seg_pos={0 0}, upload=451331]: Done begin upload segment. Dec 27 01:49:23.013369725 (514h20m22.88445277s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013382700 (514h20m22.884465735s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.013596747 (514h20m22.884679792s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014036915 (514h20m22.88511995s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.014774921 (514h20m22.885857956s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015178709 (514h20m22.886261754s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.015671768 (514h20m22.886754803s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016104583 (514h20m22.887187618s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.016573124 (514h20m22.887656169s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.017594740 (514h20m22.888677775s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.018155288 (514h20m22.889238333s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.019076802 (514h20m22.890159847s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020007875 (514h20m22.89109093s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.020871730 (514h20m22.891954775s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.021955734 (514h20m22.893038779s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:23.023529220 (514h20m22.894612265s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Uploading piece... Dec 27 01:49:24.646694782 (514h20m24.517777817s) [upload=451331]: Next returned nil segment Dec 27 01:49:24.646715311 (514h20m24.517798346s) [upload=451331]: Waiting for error group managing segments... Dec 27 01:49:25.026519311 (514h20m24.897602356s) [noise=true , address=145.79.227.23:4545 , piece_id=K22PU4BSVO3YEWF6ADV3RO4S7YLD5GZQHPPSDKI4UNAM7SKMUZHQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032030675 (514h20m24.90311372s) [noise=true , address=145.79.227.41:4085 , piece_id=SFLK2H2TEOHW3QR7L6XDMLBUIFWM6JZGP2HLWK6KHGRJ3XEXAVGQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.032385070 (514h20m24.903468105s) [noise=true , address=145.79.227.42:4122 , piece_id=BU4BADMMMVCSO7AQCR62SYFHCJ6BKRS556EIT7FTSZPWWVGOIG2Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.034087902 (514h20m24.905170947s) [noise=true , address=145.79.227.45:4230 , piece_id=XTWRYXKL43CSDVIFFYKA6OUEARUNY5T6GFULMR6DNHJONOAH632Q, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.036163545 (514h20m24.90724659s) [noise=true , address=145.79.227.11:4731 , piece_id=RUDSQNI4BSDDJJPXUYK5PPJG5W7S2EASNDN4RIHMUVP5PMG4Z5JA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.037637121 (514h20m24.908720166s) [noise=true , address=145.79.227.39:4014 , piece_id=P5BHUW47OQKRK3RYKSFENKDV4EBXOFSQ3LRVD74ZJI54JZTIQHAA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.042262888 (514h20m24.913345923s) [noise=true , address=145.79.227.37:4566 , piece_id=VAAHLQDKUJCCPGLPUCRWAXYQSXMZ2322QSSR77TNESF3CMIR6SFQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.055528678 (514h20m24.926611723s) [noise=true , address=145.79.227.15:4874 , piece_id=R7NPHNVSXPROOM4BEUT2FNNVO2CPR32KRSJM6LNTKUVOMCXYXQ4A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.199505079 (514h20m25.070588124s) [noise=true , address=77.90.31.15:4700 , piece_id=57GJ2RE62MYDVI4ED7HTLFHSGTX7GVAPVYB7CRSMJ2E2PDIUFHTA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.208923554 (514h20m25.080006589s) [noise=true , address=77.90.31.13:4627 , piece_id=U76AT63QYYFYXLT7ZJMZ6NDKDUYZDOXD7ULGVQP5FQVOIPF6BUCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.250704229 (514h20m25.121787265s) [noise=true , address=45.91.116.14:4192 , piece_id=Q5NPP7GEIQCI5Y27R34V54U2F47TMD3RMP33QXZPVIRVHULE675A, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.251548908 (514h20m25.122631943s) [noise=true , address=45.91.116.9:4263 , piece_id=2TOZI34WKTN5X6BIOYXMYPNEUEQKKUYFPF3PGDYWRA7AOOYHYXRA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304930404 (514h20m25.176013439s) [noise=true , address=45.91.116.10:4082 , piece_id=ULN6GKV4WLY74LKB3WU7GN7DNUJ5TVIPQUHUD6QLDK3BDZWRWRQA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: <nil> Dec 27 01:49:25.304949670 (514h20m25.176032705s) [seg_pos={0 0}, upload=451331]: Segment reached optimal threshold of 13 pieces. Dec 27 01:49:25.305125835 (514h20m25.17620887s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46538->145.79.227.4:4185: use of closed network connection Dec 27 01:49:25.305132128 (514h20m25.176215163s) [noise=true , address=145.79.227.4:4185 , piece_id=TYWRAEIWW3I3VA7YMEXAS7NG7KHALUOMNSZHQ2LZY5BDGKXIMDCA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305398585 (514h20m25.17648162s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305409256 (514h20m25.176492291s) [noise=true , address=77.90.31.17:4770 , piece_id=FNEQDS6C2HVPIP4NEP2ABTK2IT7DYWRUJZ5YGGZW2MJ33CO6CAFA, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305430736 (514h20m25.176513771s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 01:49:25.305439202 (514h20m25.176522257s) [noise=true , address=145.79.227.40:4050 , piece_id=ZVXNP26HZDJOYOEE5DFBNQNODFL5R5R5IYPVYFWOKZIJOOCQAVTQ, seg_pos={0 0}, upload=451331]: Long tail cancellation due to optimal threshold Dec 27 01:49:25.305457618 (514h20m25.176540653s) [seg_pos={0 0}, upload=451331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:49:25.305543511 (514h20m25.176626556s) [upload=451331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:49:25.305550996 (514h20m25.176634031s) [upload=451331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:49:25.605940909 (514h20m25.477023954s) [upload=451331]: Done uploading segments... Dec 27 01:49:25.605963783 (514h20m25.477046838s) [upload=451331]: Upload finished. err: <nil> Duration: 30.003425905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...X] duration: 30.003547025s trace: https://jaeger.dev.storj.io/trace/13afe4e986b8541b FAIL (2025-12-27 01:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:48:10.204569980 (514h19m10.075653025s) [upload=451315]: Starting upload Dec 27 01:48:10.204639803 (514h19m10.075722848s) [upload=451315]: Done starting upload Dec 27 01:48:10.204741407 (514h19m10.075824452s) [upload=451315]: Uploading segments... Dec 27 01:48:10.204870181 (514h19m10.075953226s) [upload=451315]: Got next segment. Inline: false Dec 27 01:48:10.204918564 (514h19m10.076001609s) [upload=451315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:48:10.204926339 (514h19m10.076009374s) [upload=451315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:48:10.451743432 (514h19m10.322826467s) [seg_pos={0 0}, upload=451315]: Begin upload segment... Dec 27 01:48:10.451799980 (514h19m10.322883035s) [seg_pos={0 0}, upload=451315]: Done begin upload segment. Dec 27 01:48:10.451825569 (514h19m10.322908604s) [noise=true , address=45.91.116.14:4190 , piece_id=KUR7BKI4E7FGWP5GMV4QNQTWIM2ZNWEIM5N3BV3MN3J7L2ZJ4NZA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.451849423 (514h19m10.322932459s) [noise=true , address=45.91.116.7:4011 , piece_id=XQT6RQ3EZOYXSWHSIW37ZGWLDJQITQFGK5QVBBGT3FBKDBPTZW6Q, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.452485395 (514h19m10.32356844s) [noise=true , address=145.79.227.51:4411 , piece_id=OPZVKKGIL2BATOOFS5WH324AVEWDCCK6GU6XHQFKN44O5IK6XJWA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.452725933 (514h19m10.323808978s) [noise=true , address=145.79.227.29:4942 , piece_id=HQWQAU3QQHC3YBGGGNW35BMMKU3ID5YPNFTG35GQFEXEK5QRONXA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.453129501 (514h19m10.324212546s) [noise=true , address=45.91.116.5:4300 , piece_id=4SOZGL4BLGZUY5O55VC5AKHJ7GRTZQGNA7ENUJWW4TRWLKEUJAVQ, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.453453859 (514h19m10.324536894s) [noise=true , address=45.91.116.8:4047 , piece_id=FWRKVXJGU4T4V4PKFADXVP24TFDCQKWBS5HS24ALUPOF2OSMQLOA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.453942290 (514h19m10.325025335s) [noise=true , address=145.79.227.45:4230 , piece_id=RCF344QKPBDBRYE5663MO4K6CVTT67CJFUF5YPYIHQOFCVP3SIMA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.454400883 (514h19m10.325483928s) [noise=true , address=145.79.227.14:4838 , piece_id=PVG5RGDW3VZ5FARBK544T5M6ZSBWCWZI7IABRXHCGL4UKCEPSWGQ, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.454699120 (514h19m10.325782155s) [noise=true , address=145.79.227.40:4050 , piece_id=HIPSKZIMBCOYADI2GYTQDQPFLIPSHJPPGFBXBNPXWGP6A4LG3J4A, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.455521326 (514h19m10.326604371s) [noise=true , address=145.79.227.2:4114 , piece_id=KFXTMFPZO744FMLJSBEBFW2API22PPVP46XLR3OPD4253UO4EUKA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.455599024 (514h19m10.326682069s) [noise=true , address=45.91.116.13:4156 , piece_id=O7L4NPWRGVEC3OZSPHURLLPFJVMSPZYC4ZHH6FIRRXKTUYCSFV6Q, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.456644074 (514h19m10.327727149s) [noise=true , address=45.91.116.9:4262 , piece_id=KMUXR4R2R6JSXF7DSBZZ3EIHTS2AR42NLRMGOH2WWHDABVFEIC6A, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.458175770 (514h19m10.329258815s) [noise=true , address=145.79.227.15:4874 , piece_id=DQGV2REJU6JRQJWSY2QNBUMX23OOSUTH37PBMYEXWWF3KUPFF7QQ, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.459513588 (514h19m10.330596623s) [noise=true , address=45.91.116.6:4973 , piece_id=K3N7NXRK6KBTS6NU7MJZITULBMZBLAS5AAMYPHJIILR52CS5HYSA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.460668728 (514h19m10.331751773s) [noise=true , address=145.79.227.38:4979 , piece_id=SI24EO5JVFBW6EW57CFRCV7CCHHP2MNTMS2UW24HC6QFNFAONHTQ, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:10.461224406 (514h19m10.332307451s) [noise=true , address=145.79.227.49:4338 , piece_id=ZR5LEAOWEXAPV52HNENJISY4W2GJYXFYMAUFR7B333ACDL73NGYA, seg_pos={0 0}, upload=451315]: Uploading piece... Dec 27 01:48:12.164937802 (514h19m12.036020847s) [upload=451315]: Next returned nil segment Dec 27 01:48:12.164954804 (514h19m12.036037839s) [upload=451315]: Waiting for error group managing segments... Dec 27 01:48:12.444613433 (514h19m12.315696478s) [noise=true , address=145.79.227.51:4411 , piece_id=OPZVKKGIL2BATOOFS5WH324AVEWDCCK6GU6XHQFKN44O5IK6XJWA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.461008682 (514h19m12.332091727s) [noise=true , address=145.79.227.45:4230 , piece_id=RCF344QKPBDBRYE5663MO4K6CVTT67CJFUF5YPYIHQOFCVP3SIMA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.462582208 (514h19m12.333665243s) [noise=true , address=145.79.227.15:4874 , piece_id=DQGV2REJU6JRQJWSY2QNBUMX23OOSUTH37PBMYEXWWF3KUPFF7QQ, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.468564259 (514h19m12.339647294s) [noise=true , address=145.79.227.49:4338 , piece_id=ZR5LEAOWEXAPV52HNENJISY4W2GJYXFYMAUFR7B333ACDL73NGYA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.474710793 (514h19m12.345793858s) [noise=true , address=145.79.227.38:4979 , piece_id=SI24EO5JVFBW6EW57CFRCV7CCHHP2MNTMS2UW24HC6QFNFAONHTQ, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.477724721 (514h19m12.348807766s) [noise=true , address=145.79.227.2:4114 , piece_id=KFXTMFPZO744FMLJSBEBFW2API22PPVP46XLR3OPD4253UO4EUKA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.490785080 (514h19m12.361868125s) [noise=true , address=145.79.227.14:4838 , piece_id=PVG5RGDW3VZ5FARBK544T5M6ZSBWCWZI7IABRXHCGL4UKCEPSWGQ, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.492294774 (514h19m12.363377819s) [noise=true , address=145.79.227.29:4942 , piece_id=HQWQAU3QQHC3YBGGGNW35BMMKU3ID5YPNFTG35GQFEXEK5QRONXA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.661371119 (514h19m12.532454164s) [noise=true , address=45.91.116.5:4300 , piece_id=4SOZGL4BLGZUY5O55VC5AKHJ7GRTZQGNA7ENUJWW4TRWLKEUJAVQ, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.686626319 (514h19m12.557709364s) [noise=true , address=45.91.116.8:4047 , piece_id=FWRKVXJGU4T4V4PKFADXVP24TFDCQKWBS5HS24ALUPOF2OSMQLOA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.707196116 (514h19m12.578279161s) [noise=true , address=45.91.116.14:4190 , piece_id=KUR7BKI4E7FGWP5GMV4QNQTWIM2ZNWEIM5N3BV3MN3J7L2ZJ4NZA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.724245591 (514h19m12.595328636s) [noise=true , address=45.91.116.7:4011 , piece_id=XQT6RQ3EZOYXSWHSIW37ZGWLDJQITQFGK5QVBBGT3FBKDBPTZW6Q, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.768272996 (514h19m12.639356031s) [noise=true , address=45.91.116.6:4973 , piece_id=K3N7NXRK6KBTS6NU7MJZITULBMZBLAS5AAMYPHJIILR52CS5HYSA, seg_pos={0 0}, upload=451315]: Done uploading piece. err: <nil> Dec 27 01:48:12.768298024 (514h19m12.639381059s) [seg_pos={0 0}, upload=451315]: Segment reached optimal threshold of 13 pieces. Dec 27 01:48:12.768510558 (514h19m12.639593603s) [noise=true , address=45.91.116.9:4262 , piece_id=KMUXR4R2R6JSXF7DSBZZ3EIHTS2AR42NLRMGOH2WWHDABVFEIC6A, seg_pos={0 0}, upload=451315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 01:48:12.768518914 (514h19m12.639601959s) [noise=true , address=45.91.116.9:4262 , piece_id=KMUXR4R2R6JSXF7DSBZZ3EIHTS2AR42NLRMGOH2WWHDABVFEIC6A, seg_pos={0 0}, upload=451315]: Long tail cancellation due to optimal threshold Dec 27 01:48:12.768617522 (514h19m12.639700557s) [noise=true , address=45.91.116.13:4156 , piece_id=O7L4NPWRGVEC3OZSPHURLLPFJVMSPZYC4ZHH6FIRRXKTUYCSFV6Q, seg_pos={0 0}, upload=451315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 01:48:12.768643441 (514h19m12.639726476s) [noise=true , address=45.91.116.13:4156 , piece_id=O7L4NPWRGVEC3OZSPHURLLPFJVMSPZYC4ZHH6FIRRXKTUYCSFV6Q, seg_pos={0 0}, upload=451315]: Long tail cancellation due to optimal threshold Dec 27 01:48:12.768813875 (514h19m12.63989692s) [noise=true , address=145.79.227.40:4050 , piece_id=HIPSKZIMBCOYADI2GYTQDQPFLIPSHJPPGFBXBNPXWGP6A4LG3J4A, seg_pos={0 0}, upload=451315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 01:48:12.768823955 (514h19m12.63990699s) [noise=true , address=145.79.227.40:4050 , piece_id=HIPSKZIMBCOYADI2GYTQDQPFLIPSHJPPGFBXBNPXWGP6A4LG3J4A, seg_pos={0 0}, upload=451315]: Long tail cancellation due to optimal threshold Dec 27 01:48:12.768845876 (514h19m12.639928911s) [seg_pos={0 0}, upload=451315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:48:12.768940236 (514h19m12.640023271s) [upload=451315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:48:12.768945516 (514h19m12.640028551s) [upload=451315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:48:13.058015315 (514h19m12.92909836s) [upload=451315]: Done uploading segments... Dec 27 01:48:13.058034552 (514h19m12.929117597s) [upload=451315]: Upload finished. err: <nil> Duration: 30.003568647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X...] duration: 6.168133339s trace: https://jaeger.dev.storj.io/trace/adf1c09b8e600b1 FAIL (2025-12-27 01:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ) context deadline exceeded ======LOGS====== 27 01:45:34.942938743 (514h16m34.814021789s) [upload=451278]: Starting upload Dec 27 01:45:34.943034876 (514h16m34.814117911s) [upload=451278]: Done starting upload Dec 27 01:45:34.943132332 (514h16m34.814215367s) [upload=451278]: Uploading segments... Dec 27 01:45:34.943249636 (514h16m34.814332671s) [upload=451278]: Got next segment. Inline: false Dec 27 01:45:34.943284642 (514h16m34.814367677s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:45:34.943289561 (514h16m34.814372606s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:45:35.190165026 (514h16m35.061248071s) [seg_pos={0 0}, upload=451278]: Begin upload segment... Dec 27 01:45:35.190238095 (514h16m35.06132114s) [seg_pos={0 0}, upload=451278]: Done begin upload segment. Dec 27 01:45:35.190280196 (514h16m35.061363231s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190295334 (514h16m35.061378369s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190630693 (514h16m35.061713728s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190950342 (514h16m35.062033377s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191328602 (514h16m35.062411637s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191752008 (514h16m35.062835043s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191763951 (514h16m35.062846986s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192151679 (514h16m35.063234724s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192570787 (514h16m35.063653822s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193023990 (514h16m35.064107055s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193641406 (514h16m35.064724441s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.194217964 (514h16m35.065300999s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.195531364 (514h16m35.066614399s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.196634575 (514h16m35.06771761s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.201254582 (514h16m35.072337617s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.202521705 (514h16m35.07360475s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:36.899010473 (514h16m36.770093518s) [upload=451278]: Next returned nil segment Dec 27 01:45:36.899053706 (514h16m36.770136741s) [upload=451278]: Waiting for error group managing segments... Dec 27 01:45:37.181422125 (514h16m37.05250516s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.191104982 (514h16m37.062188017s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.195162659 (514h16m37.066245694s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.204811191 (514h16m37.075894226s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.213598132 (514h16m37.084681187s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.220103971 (514h16m37.091187006s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.225515284 (514h16m37.096598319s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.226616471 (514h16m37.097699506s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.227723348 (514h16m37.098806384s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.474042615 (514h16m37.34512565s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.773256349 (514h16m37.644339394s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.933670940 (514h16m37.804753975s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163644883 (514h16m38.034727918s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163668677 (514h16m38.034751722s) [seg_pos={0 0}, upload=451278]: Segment reached optimal threshold of 13 pieces. Dec 27 01:45:38.163863569 (514h16m38.034946614s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.163873097 (514h16m38.034956142s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.163881833 (514h16m38.034964868s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection Dec 27 01:45:38.163890170 (514h16m38.034973205s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164015408 (514h16m38.035098443s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.164022151 (514h16m38.035105196s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164038412 (514h16m38.035121447s) [seg_pos={0 0}, upload=451278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:45:38.164108836 (514h16m38.035191881s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:45:38.164114036 (514h16m38.035197081s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:45:38.467060307 (514h16m38.338143352s) [upload=451278]: Done uploading segments... Dec 27 01:45:38.467078542 (514h16m38.338161577s) [upload=451278]: Upload finished. err: <nil> Duration: 6.168159778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X..] duration: 17.077422074s trace: https://jaeger.dev.storj.io/trace/adf1c09b8e600b1 FAIL (2025-12-27 01:47:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ) context deadline exceeded ======LOGS====== 27 01:45:34.942938743 (514h16m34.814021789s) [upload=451278]: Starting upload Dec 27 01:45:34.943034876 (514h16m34.814117911s) [upload=451278]: Done starting upload Dec 27 01:45:34.943132332 (514h16m34.814215367s) [upload=451278]: Uploading segments... Dec 27 01:45:34.943249636 (514h16m34.814332671s) [upload=451278]: Got next segment. Inline: false Dec 27 01:45:34.943284642 (514h16m34.814367677s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:45:34.943289561 (514h16m34.814372606s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:45:35.190165026 (514h16m35.061248071s) [seg_pos={0 0}, upload=451278]: Begin upload segment... Dec 27 01:45:35.190238095 (514h16m35.06132114s) [seg_pos={0 0}, upload=451278]: Done begin upload segment. Dec 27 01:45:35.190280196 (514h16m35.061363231s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190295334 (514h16m35.061378369s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190630693 (514h16m35.061713728s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190950342 (514h16m35.062033377s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191328602 (514h16m35.062411637s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191752008 (514h16m35.062835043s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191763951 (514h16m35.062846986s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192151679 (514h16m35.063234724s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192570787 (514h16m35.063653822s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193023990 (514h16m35.064107055s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193641406 (514h16m35.064724441s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.194217964 (514h16m35.065300999s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.195531364 (514h16m35.066614399s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.196634575 (514h16m35.06771761s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.201254582 (514h16m35.072337617s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.202521705 (514h16m35.07360475s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:36.899010473 (514h16m36.770093518s) [upload=451278]: Next returned nil segment Dec 27 01:45:36.899053706 (514h16m36.770136741s) [upload=451278]: Waiting for error group managing segments... Dec 27 01:45:37.181422125 (514h16m37.05250516s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.191104982 (514h16m37.062188017s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.195162659 (514h16m37.066245694s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.204811191 (514h16m37.075894226s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.213598132 (514h16m37.084681187s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.220103971 (514h16m37.091187006s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.225515284 (514h16m37.096598319s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.226616471 (514h16m37.097699506s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.227723348 (514h16m37.098806384s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.474042615 (514h16m37.34512565s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.773256349 (514h16m37.644339394s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.933670940 (514h16m37.804753975s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163644883 (514h16m38.034727918s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163668677 (514h16m38.034751722s) [seg_pos={0 0}, upload=451278]: Segment reached optimal threshold of 13 pieces. Dec 27 01:45:38.163863569 (514h16m38.034946614s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.163873097 (514h16m38.034956142s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.163881833 (514h16m38.034964868s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection Dec 27 01:45:38.163890170 (514h16m38.034973205s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164015408 (514h16m38.035098443s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.164022151 (514h16m38.035105196s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164038412 (514h16m38.035121447s) [seg_pos={0 0}, upload=451278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:45:38.164108836 (514h16m38.035191881s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:45:38.164114036 (514h16m38.035197081s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:45:38.467060307 (514h16m38.338143352s) [upload=451278]: Done uploading segments... Dec 27 01:45:38.467078542 (514h16m38.338161577s) [upload=451278]: Upload finished. err: <nil> Duration: 17.077465857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X.] duration: 5.956249184s trace: https://jaeger.dev.storj.io/trace/adf1c09b8e600b1 FAIL (2025-12-27 01:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ) context deadline exceeded ======LOGS====== 27 01:45:34.942938743 (514h16m34.814021789s) [upload=451278]: Starting upload Dec 27 01:45:34.943034876 (514h16m34.814117911s) [upload=451278]: Done starting upload Dec 27 01:45:34.943132332 (514h16m34.814215367s) [upload=451278]: Uploading segments... Dec 27 01:45:34.943249636 (514h16m34.814332671s) [upload=451278]: Got next segment. Inline: false Dec 27 01:45:34.943284642 (514h16m34.814367677s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:45:34.943289561 (514h16m34.814372606s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:45:35.190165026 (514h16m35.061248071s) [seg_pos={0 0}, upload=451278]: Begin upload segment... Dec 27 01:45:35.190238095 (514h16m35.06132114s) [seg_pos={0 0}, upload=451278]: Done begin upload segment. Dec 27 01:45:35.190280196 (514h16m35.061363231s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190295334 (514h16m35.061378369s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190630693 (514h16m35.061713728s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190950342 (514h16m35.062033377s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191328602 (514h16m35.062411637s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191752008 (514h16m35.062835043s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191763951 (514h16m35.062846986s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192151679 (514h16m35.063234724s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192570787 (514h16m35.063653822s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193023990 (514h16m35.064107055s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193641406 (514h16m35.064724441s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.194217964 (514h16m35.065300999s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.195531364 (514h16m35.066614399s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.196634575 (514h16m35.06771761s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.201254582 (514h16m35.072337617s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.202521705 (514h16m35.07360475s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:36.899010473 (514h16m36.770093518s) [upload=451278]: Next returned nil segment Dec 27 01:45:36.899053706 (514h16m36.770136741s) [upload=451278]: Waiting for error group managing segments... Dec 27 01:45:37.181422125 (514h16m37.05250516s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.191104982 (514h16m37.062188017s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.195162659 (514h16m37.066245694s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.204811191 (514h16m37.075894226s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.213598132 (514h16m37.084681187s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.220103971 (514h16m37.091187006s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.225515284 (514h16m37.096598319s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.226616471 (514h16m37.097699506s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.227723348 (514h16m37.098806384s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.474042615 (514h16m37.34512565s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.773256349 (514h16m37.644339394s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.933670940 (514h16m37.804753975s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163644883 (514h16m38.034727918s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163668677 (514h16m38.034751722s) [seg_pos={0 0}, upload=451278]: Segment reached optimal threshold of 13 pieces. Dec 27 01:45:38.163863569 (514h16m38.034946614s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.163873097 (514h16m38.034956142s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.163881833 (514h16m38.034964868s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection Dec 27 01:45:38.163890170 (514h16m38.034973205s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164015408 (514h16m38.035098443s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.164022151 (514h16m38.035105196s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164038412 (514h16m38.035121447s) [seg_pos={0 0}, upload=451278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:45:38.164108836 (514h16m38.035191881s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:45:38.164114036 (514h16m38.035197081s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:45:38.467060307 (514h16m38.338143352s) [upload=451278]: Done uploading segments... Dec 27 01:45:38.467078542 (514h16m38.338161577s) [upload=451278]: Upload finished. err: <nil> Duration: 5.956269763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.X] duration: 30.003685662s trace: https://jaeger.dev.storj.io/trace/adf1c09b8e600b1 FAIL (2025-12-27 01:46:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ) context deadline exceeded ======LOGS====== 27 01:45:34.942938743 (514h16m34.814021789s) [upload=451278]: Starting upload Dec 27 01:45:34.943034876 (514h16m34.814117911s) [upload=451278]: Done starting upload Dec 27 01:45:34.943132332 (514h16m34.814215367s) [upload=451278]: Uploading segments... Dec 27 01:45:34.943249636 (514h16m34.814332671s) [upload=451278]: Got next segment. Inline: false Dec 27 01:45:34.943284642 (514h16m34.814367677s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:45:34.943289561 (514h16m34.814372606s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:45:35.190165026 (514h16m35.061248071s) [seg_pos={0 0}, upload=451278]: Begin upload segment... Dec 27 01:45:35.190238095 (514h16m35.06132114s) [seg_pos={0 0}, upload=451278]: Done begin upload segment. Dec 27 01:45:35.190280196 (514h16m35.061363231s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190295334 (514h16m35.061378369s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190630693 (514h16m35.061713728s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.190950342 (514h16m35.062033377s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191328602 (514h16m35.062411637s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191752008 (514h16m35.062835043s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.191763951 (514h16m35.062846986s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192151679 (514h16m35.063234724s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.192570787 (514h16m35.063653822s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193023990 (514h16m35.064107055s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.193641406 (514h16m35.064724441s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.194217964 (514h16m35.065300999s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.195531364 (514h16m35.066614399s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.196634575 (514h16m35.06771761s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.201254582 (514h16m35.072337617s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:35.202521705 (514h16m35.07360475s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Uploading piece... Dec 27 01:45:36.899010473 (514h16m36.770093518s) [upload=451278]: Next returned nil segment Dec 27 01:45:36.899053706 (514h16m36.770136741s) [upload=451278]: Waiting for error group managing segments... Dec 27 01:45:37.181422125 (514h16m37.05250516s) [noise=true , address=145.79.227.51:4411 , piece_id=53XKRA4BKSRFPJJRF4IWJGRNGEYKG5KYST3M7SMHXKBRQKO2NTNA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.191104982 (514h16m37.062188017s) [noise=true , address=145.79.227.11:4731 , piece_id=DOAVE5IZGZUCNU3YNBAH6GIR3YVB256QAYT32DEHKDSW6JVH4PGA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.195162659 (514h16m37.066245694s) [noise=true , address=145.79.227.14:4837 , piece_id=3YLETDC57CUCCZZYWQFDMA467GIRZWVYGAAOQ7XK24G6ND6UJHLQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.204811191 (514h16m37.075894226s) [noise=true , address=145.79.227.5:4222 , piece_id=FXQSCKY4QSNQ3PEWWFFNXFZOP4GXHB4JDLJCKEHYAZWNBQ6SS4SQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.213598132 (514h16m37.084681187s) [noise=true , address=145.79.227.37:4566 , piece_id=US6UH4NEPKO2BAZWKRRUJ6AXCCP6X6XS3BFW3QUAVEF44E66LHBA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.220103971 (514h16m37.091187006s) [noise=true , address=145.79.227.52:4447 , piece_id=C7FBTGZ37ZDWTOLDXGP6E2Z2NBZDD2BJHOUQIALXNON6YITRKMPA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.225515284 (514h16m37.096598319s) [noise=true , address=145.79.227.6:4259 , piece_id=N7IEALOR3GJ6OHZ2ACDVCHPW2VDMI6VAZFC7HAWVDNPYOI4G3W2Q, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.226616471 (514h16m37.097699506s) [noise=true , address=145.79.227.50:4374 , piece_id=2EJB4TGSJB542APJQVJMR6XODXRDSDUXGQG3BL6KU35ZPBTUGAIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.227723348 (514h16m37.098806384s) [noise=true , address=145.79.227.15:4874 , piece_id=XO6XJ53WNHLK6KGJQW6HLOVTD2XW2FFL7YKDMI5N5OBWTKSE2WKA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.474042615 (514h16m37.34512565s) [noise=true , address=77.90.31.11:4554 , piece_id=MHLBKJEWQL4DAYTL7A7UBQC2WUZCGQ2AXPJH4FCJ2LWHFCRUKGWQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.773256349 (514h16m37.644339394s) [noise=true , address=45.91.116.7:4012 , piece_id=YJ7DQY2ULTTRTEZXN2SOJNDWEHZ3EW2HPXRLZWY5VARTYVXHGHIA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:37.933670940 (514h16m37.804753975s) [noise=true , address=45.91.116.12:4226 , piece_id=URGFZA76I7SSLAWBF4SI54G54QOBYG6NYF6CUKZT7A2Y2BCDXELQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163644883 (514h16m38.034727918s) [noise=true , address=45.91.116.14:4190 , piece_id=JAPPNPTDBJIGIDZJNOEM5JVPHHSJ7W6VSCL4XJVYHEZSJ77SC6EQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: <nil> Dec 27 01:45:38.163668677 (514h16m38.034751722s) [seg_pos={0 0}, upload=451278]: Segment reached optimal threshold of 13 pieces. Dec 27 01:45:38.163863569 (514h16m38.034946614s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.163873097 (514h16m38.034956142s) [noise=true , address=77.90.31.12:4590 , piece_id=W25ONWYBX6JE5BZXVPO6KODRTFLKR3OTR5U26QFK42FGDVZYZ2CA, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.163881833 (514h16m38.034964868s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39654->145.79.227.2:4114: use of closed network connection Dec 27 01:45:38.163890170 (514h16m38.034973205s) [noise=true , address=145.79.227.2:4114 , piece_id=J5KSEWL3SSCBSVQB43WORZBZQFNUBJXFYLW6MCLVL6IK65NGLJ6A, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164015408 (514h16m38.035098443s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 01:45:38.164022151 (514h16m38.035105196s) [noise=true , address=45.91.116.11:4117 , piece_id=K5ZXKQWTHXKS6TGK2GB6QQI5PH4LFNG5DYCULO7H55XR2H2WGTJQ, seg_pos={0 0}, upload=451278]: Long tail cancellation due to optimal threshold Dec 27 01:45:38.164038412 (514h16m38.035121447s) [seg_pos={0 0}, upload=451278]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:45:38.164108836 (514h16m38.035191881s) [upload=451278]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:45:38.164114036 (514h16m38.035197081s) [upload=451278]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:45:38.467060307 (514h16m38.338143352s) [upload=451278]: Done uploading segments... Dec 27 01:45:38.467078542 (514h16m38.338161577s) [upload=451278]: Upload finished. err: <nil> Duration: 30.003705049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX.] duration: 21.560358888s trace: https://jaeger.dev.storj.io/trace/43ee8330cdbd4bc7 FAIL (2025-12-27 01:45:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ) context deadline exceeded ======LOGS====== Dec 27 01:44:00.387648766 (514h15m0.258731802s) [upload=451259]: Starting upload Dec 27 01:44:00.387704894 (514h15m0.258787939s) [upload=451259]: Done starting upload Dec 27 01:44:00.387811477 (514h15m0.258894512s) [upload=451259]: Uploading segments... Dec 27 01:44:00.387924001 (514h15m0.259007036s) [upload=451259]: Got next segment. Inline: false Dec 27 01:44:00.387980919 (514h15m0.259063954s) [upload=451259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:44:00.387988163 (514h15m0.259071198s) [upload=451259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:44:00.637155001 (514h15m0.508238036s) [seg_pos={0 0}, upload=451259]: Begin upload segment... Dec 27 01:44:00.637200207 (514h15m0.508283242s) [seg_pos={0 0}, upload=451259]: Done begin upload segment. Dec 27 01:44:00.637219223 (514h15m0.508302258s) [noise=true , address=45.91.116.11:4120 , piece_id=DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.637267876 (514h15m0.508350911s) [noise=true , address=77.90.31.19:4842 , piece_id=K2UDWJ5HUQEHN6VQSFWKYB5KPMEATCXMYUXMDLMK5RVBOCKGSVJA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.637942080 (514h15m0.509025125s) [noise=true , address=145.79.227.37:4565 , piece_id=UAYAG3CFTRH5QLVJTXOSASX4V75JLAKAK4JTTUNVSU5LWON23GDQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638287207 (514h15m0.509370242s) [noise=true , address=145.79.227.41:4087 , piece_id=YYDX6NHXLHYGYUR2VW4MQGWIYRWYN6BQFI5DCVRZP7B23JCXO72A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638655197 (514h15m0.509738233s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638937435 (514h15m0.51002047s) [noise=true , address=145.79.227.42:4121 , piece_id=S2HCOICEF3HU2JPMOGSCOATTV2MNXPPN4GNRMEDMVZDWOCRKK47Q, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639280919 (514h15m0.510363954s) [noise=true , address=145.79.227.11:4731 , piece_id=7KHLBNYRGEW7NANRCPYMJXGY6JLN7YUMYCIJSXVU62QCFLV2AZOQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639578095 (514h15m0.51066113s) [noise=true , address=145.79.227.15:4875 , piece_id=JJ7AXT3KN2K5TGWHUJEOQ2CPMKDNG73OL47ECFUDSDGMJ3CCXQTA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639951586 (514h15m0.511034631s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.640977891 (514h15m0.512060926s) [noise=true , address=77.90.31.10:4520 , piece_id=WBDHWS6XNACLGSP266SYHAZ6CCPGDPRUFQ4Z65Z3HXAYE4MGPU2A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.641037244 (514h15m0.512120289s) [noise=true , address=145.79.227.47:4266 , piece_id=C6NTULBTTP2ICMU3OYHF4AKQEMJ2VN2VF5IGA2CLRU4LT5LDBB5A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.642090048 (514h15m0.513173093s) [noise=true , address=145.79.227.45:4229 , piece_id=COTJRNQ3FSTQCRIKTYUR5KHDNLCBJ3OEE3FH43AIHBCUPRFWXCCQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.642244131 (514h15m0.513327176s) [noise=true , address=145.79.227.50:4375 , piece_id=3GAYYEBD7FC6I4XIDMZ3USFAPA7GF2DDN4KNCZUWRYE7IUDZGYRA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.643488231 (514h15m0.514571276s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.643771510 (514h15m0.514854555s) [noise=true , address=145.79.227.52:4447 , piece_id=CTD34ALBRTJ3FCHDNDSJA7BMDOJZHGHED7425MBA4HNB2V65QFJQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.644859612 (514h15m0.515942647s) [noise=true , address=45.91.116.8:4047 , piece_id=EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:02.309830688 (514h15m2.180913733s) [upload=451259]: Next returned nil segment Dec 27 01:44:02.309974952 (514h15m2.181058017s) [upload=451259]: Waiting for error group managing segments... Dec 27 01:44:02.647328441 (514h15m2.518411486s) [noise=true , address=145.79.227.41:4087 , piece_id=YYDX6NHXLHYGYUR2VW4MQGWIYRWYN6BQFI5DCVRZP7B23JCXO72A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.648521883 (514h15m2.519604928s) [noise=true , address=145.79.227.52:4447 , piece_id=CTD34ALBRTJ3FCHDNDSJA7BMDOJZHGHED7425MBA4HNB2V65QFJQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.649133617 (514h15m2.520216652s) [noise=true , address=145.79.227.42:4121 , piece_id=S2HCOICEF3HU2JPMOGSCOATTV2MNXPPN4GNRMEDMVZDWOCRKK47Q, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.649863236 (514h15m2.520946281s) [noise=true , address=145.79.227.15:4875 , piece_id=JJ7AXT3KN2K5TGWHUJEOQ2CPMKDNG73OL47ECFUDSDGMJ3CCXQTA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.654546905 (514h15m2.52562994s) [noise=true , address=145.79.227.50:4375 , piece_id=3GAYYEBD7FC6I4XIDMZ3USFAPA7GF2DDN4KNCZUWRYE7IUDZGYRA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.666659158 (514h15m2.537742203s) [noise=true , address=145.79.227.11:4731 , piece_id=7KHLBNYRGEW7NANRCPYMJXGY6JLN7YUMYCIJSXVU62QCFLV2AZOQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.678310114 (514h15m2.549393149s) [noise=true , address=145.79.227.37:4565 , piece_id=UAYAG3CFTRH5QLVJTXOSASX4V75JLAKAK4JTTUNVSU5LWON23GDQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.685293561 (514h15m2.556376606s) [noise=true , address=145.79.227.47:4266 , piece_id=C6NTULBTTP2ICMU3OYHF4AKQEMJ2VN2VF5IGA2CLRU4LT5LDBB5A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.704543676 (514h15m2.575626721s) [noise=true , address=145.79.227.45:4229 , piece_id=COTJRNQ3FSTQCRIKTYUR5KHDNLCBJ3OEE3FH43AIHBCUPRFWXCCQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.884625045 (514h15m2.755708091s) [noise=true , address=77.90.31.19:4842 , piece_id=K2UDWJ5HUQEHN6VQSFWKYB5KPMEATCXMYUXMDLMK5RVBOCKGSVJA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.956455731 (514h15m2.827538786s) [noise=true , address=45.91.116.11:4120 , piece_id=DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.012201839 (514h15m2.883284884s) [noise=true , address=45.91.116.8:4047 , piece_id=EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.510298845 (514h15m3.3813819s) [noise=true , address=77.90.31.10:4520 , piece_id=WBDHWS6XNACLGSP266SYHAZ6CCPGDPRUFQ4Z65Z3HXAYE4MGPU2A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.510327550 (514h15m3.381410605s) [seg_pos={0 0}, upload=451259]: Segment reached optimal threshold of 13 pieces. Dec 27 01:44:03.510800289 (514h15m3.381883334s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.510812112 (514h15m3.381895148s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.510831769 (514h15m3.381914814s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.510844424 (514h15m3.381927459s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.511043553 (514h15m3.382126608s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.511054965 (514h15m3.38213801s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.511112454 (514h15m3.382195509s) [seg_pos={0 0}, upload=451259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:44:03.511221441 (514h15m3.382304487s) [upload=451259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:44:03.511228686 (514h15m3.382311731s) [upload=451259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:44:03.809700728 (514h15m3.680783773s) [upload=451259]: Done uploading segments... Dec 27 01:44:03.809725254 (514h15m3.680808289s) [upload=451259]: Upload finished. err: <nil> Duration: 21.56053917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.XX] duration: 30.003777988s trace: https://jaeger.dev.storj.io/trace/43ee8330cdbd4bc7 FAIL (2025-12-27 01:44:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ) context deadline exceeded ======LOGS====== Dec 27 01:44:00.387648766 (514h15m0.258731802s) [upload=451259]: Starting upload Dec 27 01:44:00.387704894 (514h15m0.258787939s) [upload=451259]: Done starting upload Dec 27 01:44:00.387811477 (514h15m0.258894512s) [upload=451259]: Uploading segments... Dec 27 01:44:00.387924001 (514h15m0.259007036s) [upload=451259]: Got next segment. Inline: false Dec 27 01:44:00.387980919 (514h15m0.259063954s) [upload=451259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:44:00.387988163 (514h15m0.259071198s) [upload=451259]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:44:00.637155001 (514h15m0.508238036s) [seg_pos={0 0}, upload=451259]: Begin upload segment... Dec 27 01:44:00.637200207 (514h15m0.508283242s) [seg_pos={0 0}, upload=451259]: Done begin upload segment. Dec 27 01:44:00.637219223 (514h15m0.508302258s) [noise=true , address=45.91.116.11:4120 , piece_id=DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.637267876 (514h15m0.508350911s) [noise=true , address=77.90.31.19:4842 , piece_id=K2UDWJ5HUQEHN6VQSFWKYB5KPMEATCXMYUXMDLMK5RVBOCKGSVJA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.637942080 (514h15m0.509025125s) [noise=true , address=145.79.227.37:4565 , piece_id=UAYAG3CFTRH5QLVJTXOSASX4V75JLAKAK4JTTUNVSU5LWON23GDQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638287207 (514h15m0.509370242s) [noise=true , address=145.79.227.41:4087 , piece_id=YYDX6NHXLHYGYUR2VW4MQGWIYRWYN6BQFI5DCVRZP7B23JCXO72A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638655197 (514h15m0.509738233s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.638937435 (514h15m0.51002047s) [noise=true , address=145.79.227.42:4121 , piece_id=S2HCOICEF3HU2JPMOGSCOATTV2MNXPPN4GNRMEDMVZDWOCRKK47Q, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639280919 (514h15m0.510363954s) [noise=true , address=145.79.227.11:4731 , piece_id=7KHLBNYRGEW7NANRCPYMJXGY6JLN7YUMYCIJSXVU62QCFLV2AZOQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639578095 (514h15m0.51066113s) [noise=true , address=145.79.227.15:4875 , piece_id=JJ7AXT3KN2K5TGWHUJEOQ2CPMKDNG73OL47ECFUDSDGMJ3CCXQTA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.639951586 (514h15m0.511034631s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.640977891 (514h15m0.512060926s) [noise=true , address=77.90.31.10:4520 , piece_id=WBDHWS6XNACLGSP266SYHAZ6CCPGDPRUFQ4Z65Z3HXAYE4MGPU2A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.641037244 (514h15m0.512120289s) [noise=true , address=145.79.227.47:4266 , piece_id=C6NTULBTTP2ICMU3OYHF4AKQEMJ2VN2VF5IGA2CLRU4LT5LDBB5A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.642090048 (514h15m0.513173093s) [noise=true , address=145.79.227.45:4229 , piece_id=COTJRNQ3FSTQCRIKTYUR5KHDNLCBJ3OEE3FH43AIHBCUPRFWXCCQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.642244131 (514h15m0.513327176s) [noise=true , address=145.79.227.50:4375 , piece_id=3GAYYEBD7FC6I4XIDMZ3USFAPA7GF2DDN4KNCZUWRYE7IUDZGYRA, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.643488231 (514h15m0.514571276s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.643771510 (514h15m0.514854555s) [noise=true , address=145.79.227.52:4447 , piece_id=CTD34ALBRTJ3FCHDNDSJA7BMDOJZHGHED7425MBA4HNB2V65QFJQ, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:00.644859612 (514h15m0.515942647s) [noise=true , address=45.91.116.8:4047 , piece_id=EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A, seg_pos={0 0}, upload=451259]: Uploading piece... Dec 27 01:44:02.309830688 (514h15m2.180913733s) [upload=451259]: Next returned nil segment Dec 27 01:44:02.309974952 (514h15m2.181058017s) [upload=451259]: Waiting for error group managing segments... Dec 27 01:44:02.647328441 (514h15m2.518411486s) [noise=true , address=145.79.227.41:4087 , piece_id=YYDX6NHXLHYGYUR2VW4MQGWIYRWYN6BQFI5DCVRZP7B23JCXO72A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.648521883 (514h15m2.519604928s) [noise=true , address=145.79.227.52:4447 , piece_id=CTD34ALBRTJ3FCHDNDSJA7BMDOJZHGHED7425MBA4HNB2V65QFJQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.649133617 (514h15m2.520216652s) [noise=true , address=145.79.227.42:4121 , piece_id=S2HCOICEF3HU2JPMOGSCOATTV2MNXPPN4GNRMEDMVZDWOCRKK47Q, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.649863236 (514h15m2.520946281s) [noise=true , address=145.79.227.15:4875 , piece_id=JJ7AXT3KN2K5TGWHUJEOQ2CPMKDNG73OL47ECFUDSDGMJ3CCXQTA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.654546905 (514h15m2.52562994s) [noise=true , address=145.79.227.50:4375 , piece_id=3GAYYEBD7FC6I4XIDMZ3USFAPA7GF2DDN4KNCZUWRYE7IUDZGYRA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.666659158 (514h15m2.537742203s) [noise=true , address=145.79.227.11:4731 , piece_id=7KHLBNYRGEW7NANRCPYMJXGY6JLN7YUMYCIJSXVU62QCFLV2AZOQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.678310114 (514h15m2.549393149s) [noise=true , address=145.79.227.37:4565 , piece_id=UAYAG3CFTRH5QLVJTXOSASX4V75JLAKAK4JTTUNVSU5LWON23GDQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.685293561 (514h15m2.556376606s) [noise=true , address=145.79.227.47:4266 , piece_id=C6NTULBTTP2ICMU3OYHF4AKQEMJ2VN2VF5IGA2CLRU4LT5LDBB5A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.704543676 (514h15m2.575626721s) [noise=true , address=145.79.227.45:4229 , piece_id=COTJRNQ3FSTQCRIKTYUR5KHDNLCBJ3OEE3FH43AIHBCUPRFWXCCQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.884625045 (514h15m2.755708091s) [noise=true , address=77.90.31.19:4842 , piece_id=K2UDWJ5HUQEHN6VQSFWKYB5KPMEATCXMYUXMDLMK5RVBOCKGSVJA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:02.956455731 (514h15m2.827538786s) [noise=true , address=45.91.116.11:4120 , piece_id=DPS4CF36AP3JA7MPAGS7DLFEHZ73N6VVNMKYWM5KVN5JCOHR2ZPQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.012201839 (514h15m2.883284884s) [noise=true , address=45.91.116.8:4047 , piece_id=EXCYSDA2S5KYQMPBYIIFWWJ6RJUUNSHHDKZYYFW3ZDV26FBJR62A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.510298845 (514h15m3.3813819s) [noise=true , address=77.90.31.10:4520 , piece_id=WBDHWS6XNACLGSP266SYHAZ6CCPGDPRUFQ4Z65Z3HXAYE4MGPU2A, seg_pos={0 0}, upload=451259]: Done uploading piece. err: <nil> Dec 27 01:44:03.510327550 (514h15m3.381410605s) [seg_pos={0 0}, upload=451259]: Segment reached optimal threshold of 13 pieces. Dec 27 01:44:03.510800289 (514h15m3.381883334s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.510812112 (514h15m3.381895148s) [noise=true , address=77.90.31.21:4878 , piece_id=YYHZ224FYUIOCOPPSICXI4YP2OH25WW64W6LR77JAAL5ZJCEPYRA, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.510831769 (514h15m3.381914814s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.510844424 (514h15m3.381927459s) [noise=true , address=45.91.116.10:4081 , piece_id=G4D2SAHRC3HE6X5GRW5EYQ34DZC7WXTBH6BNGHXUBAJZTAWA2GXQ, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.511043553 (514h15m3.382126608s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 01:44:03.511054965 (514h15m3.38213801s) [noise=true , address=77.90.31.14:4663 , piece_id=C36JHBVT2LMAAEGZV2R5VJG475KIQB2JNIXMPEBI7ZU3VS4BU4CA, seg_pos={0 0}, upload=451259]: Long tail cancellation due to optimal threshold Dec 27 01:44:03.511112454 (514h15m3.382195509s) [seg_pos={0 0}, upload=451259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:44:03.511221441 (514h15m3.382304487s) [upload=451259]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:44:03.511228686 (514h15m3.382311731s) [upload=451259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:44:03.809700728 (514h15m3.680783773s) [upload=451259]: Done uploading segments... Dec 27 01:44:03.809725254 (514h15m3.680808289s) [upload=451259]: Upload finished. err: <nil> Duration: 30.003801454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.X] duration: 30.00307702s trace: https://jaeger.dev.storj.io/trace/d40f52573464652 FAIL (2025-12-27 01:43:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: LEUVZRRXNMITUVUNGPCGIFJLITDNNHHU2LLITK5HGL43TGWH3Y6A) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 2XK5SI3T3JWGSKUY4CU4Q2A37YXBUE2D2NI5WRNGZM3TOVU7UIGA) context deadline exceeded ======LOGS====== Dec 27 01:43:18.600874339 (514h14m18.471957374s) [upload=451246]: Starting upload Dec 27 01:43:18.600947448 (514h14m18.472030514s) [upload=451246]: Done starting upload Dec 27 01:43:18.601167778 (514h14m18.472250813s) [upload=451246]: Uploading segments... Dec 27 01:43:18.601215648 (514h14m18.472298694s) [upload=451246]: Got next segment. Inline: false Dec 27 01:43:18.601249473 (514h14m18.472332508s) [upload=451246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:43:18.601256878 (514h14m18.472339923s) [upload=451246]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:43:18.844140582 (514h14m18.715223627s) [seg_pos={0 0}, upload=451246]: Begin upload segment... Dec 27 01:43:18.844196489 (514h14m18.715279534s) [seg_pos={0 0}, upload=451246]: Done begin upload segment. Dec 27 01:43:18.844212278 (514h14m18.715295314s) [noise=true , address=145.79.227.11:4732 , piece_id=6QW6NG6NTCBZ2FONK5GUWVW4LJYZ7BKHQV5ZLUVBOKDWI33PJGPQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.844284105 (514h14m18.71536714s) [noise=true , address=45.91.116.12:4225 , piece_id=LEUVZRRXNMITUVUNGPCGIFJLITDNNHHU2LLITK5HGL43TGWH3Y6A, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.844728512 (514h14m18.715811547s) [noise=true , address=145.79.227.42:4122 , piece_id=IY4IHM3AU4OCAT2NE2X5GKJ2DGSLTHH53Y4YLINJW5Z3B7TCWSNA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.845113324 (514h14m18.716196369s) [noise=true , address=145.79.227.52:4446 , piece_id=K3BBKYHG2RANZO7CJNA7VR2JPO2SRFXMF4HDKNAYUW27PYF3GCPQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.845506322 (514h14m18.716589367s) [noise=true , address=45.91.116.6:4974 , piece_id=E63LNYCHIBDF4UTBF4VXOAEBVGDXQP36Z77PLHFBAOOHIW2LAEMA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.845912967 (514h14m18.716996012s) [noise=true , address=145.79.227.16:4909 , piece_id=AYOMW4SPF4XUPXVGL6BCEQTWUJTTMDZOVHIX2AVTYK4IPKFKKDDA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.846289884 (514h14m18.717372929s) [noise=true , address=145.79.227.51:4410 , piece_id=WFXRVURIAU7LE4L7GY3RSUXHDGDCSEG3INL75B4YAACN7H47FPMA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.846763095 (514h14m18.71784614s) [noise=true , address=145.79.227.3:4149 , piece_id=WMDSIKT75RZBOUPDCUEZRAGYSGCSLVH56J7XJAE774IYGTBLD5FA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.847159249 (514h14m18.718242294s) [noise=true , address=45.91.116.10:4083 , piece_id=2CMQOAF3QDWT4UDWXXEVDWYMQZWK6MVXUIN26AVEL3UJ63KLBFHA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.847485822 (514h14m18.718568867s) [noise=true , address=77.90.31.10:4519 , piece_id=USHS6FX4NDTQ7Q6IBV73K64YE2V3TFGPFM54VGOI6SWJT66WMUPQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.847900231 (514h14m18.718983296s) [noise=true , address=145.79.227.46:4601 , piece_id=AY563LMAY54FSQCUYDRTJP2SQSEXEZ23K74GP722TZPV7CHMHZWQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.848329117 (514h14m18.719412172s) [noise=true , address=77.90.31.17:4773 , piece_id=2XK5SI3T3JWGSKUY4CU4Q2A37YXBUE2D2NI5WRNGZM3TOVU7UIGA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.848520772 (514h14m18.719603817s) [noise=true , address=145.79.227.50:4373 , piece_id=QL3PKIXJWWAF7ESCR7NG54W2PWB64GEY56APZMWTFUIZXVNO4OTA, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.848950471 (514h14m18.720033516s) [noise=true , address=45.91.116.8:4045 , piece_id=BENM567HPTFMNSFUP2PH6JN56FD2OH6B7KTCBWJT665DH4NSKALQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.849360390 (514h14m18.720443436s) [noise=true , address=145.79.227.38:4977 , piece_id=7OH6NWZD66EO5BV4UQAZ36AW3YQJF6DN3GYYFZEB7KKPHPIHZ4LQ, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:18.849770872 (514h14m18.720853937s) [noise=true , address=145.79.227.36:4518 , piece_id=SRW7MW2GHIEGMNSXFN57IR4X2RB42ENQ753HLQW4TYF7424S2W7Q, seg_pos={0 0}, upload=451246]: Uploading piece... Dec 27 01:43:20.548122147 (514h14m20.419205182s) [upload=451246]: Next returned nil segment Dec 27 01:43:20.548150441 (514h14m20.419233476s) [upload=451246]: Waiting for error group managing segments... Dec 27 01:43:20.847176519 (514h14m20.718259564s) [noise=true , address=145.79.227.51:4410 , piece_id=WFXRVURIAU7LE4L7GY3RSUXHDGDCSEG3INL75B4YAACN7H47FPMA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.853676435 (514h14m20.72475947s) [noise=true , address=145.79.227.52:4446 , piece_id=K3BBKYHG2RANZO7CJNA7VR2JPO2SRFXMF4HDKNAYUW27PYF3GCPQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.853769893 (514h14m20.724852938s) [noise=true , address=145.79.227.46:4601 , piece_id=AY563LMAY54FSQCUYDRTJP2SQSEXEZ23K74GP722TZPV7CHMHZWQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.856252851 (514h14m20.727335886s) [noise=true , address=145.79.227.38:4977 , piece_id=7OH6NWZD66EO5BV4UQAZ36AW3YQJF6DN3GYYFZEB7KKPHPIHZ4LQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.859378393 (514h14m20.730461428s) [noise=true , address=145.79.227.16:4909 , piece_id=AYOMW4SPF4XUPXVGL6BCEQTWUJTTMDZOVHIX2AVTYK4IPKFKKDDA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.866835150 (514h14m20.737918185s) [noise=true , address=145.79.227.11:4732 , piece_id=6QW6NG6NTCBZ2FONK5GUWVW4LJYZ7BKHQV5ZLUVBOKDWI33PJGPQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.869329460 (514h14m20.740412495s) [noise=true , address=145.79.227.42:4122 , piece_id=IY4IHM3AU4OCAT2NE2X5GKJ2DGSLTHH53Y4YLINJW5Z3B7TCWSNA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.869725906 (514h14m20.740808941s) [noise=true , address=145.79.227.3:4149 , piece_id=WMDSIKT75RZBOUPDCUEZRAGYSGCSLVH56J7XJAE774IYGTBLD5FA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.876399453 (514h14m20.747482488s) [noise=true , address=145.79.227.36:4518 , piece_id=SRW7MW2GHIEGMNSXFN57IR4X2RB42ENQ753HLQW4TYF7424S2W7Q, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:20.880593378 (514h14m20.751676413s) [noise=true , address=145.79.227.50:4373 , piece_id=QL3PKIXJWWAF7ESCR7NG54W2PWB64GEY56APZMWTFUIZXVNO4OTA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:21.115480917 (514h14m20.986563962s) [noise=true , address=45.91.116.10:4083 , piece_id=2CMQOAF3QDWT4UDWXXEVDWYMQZWK6MVXUIN26AVEL3UJ63KLBFHA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:21.118763728 (514h14m20.989846773s) [noise=true , address=77.90.31.17:4773 , piece_id=2XK5SI3T3JWGSKUY4CU4Q2A37YXBUE2D2NI5WRNGZM3TOVU7UIGA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:21.124119396 (514h14m20.995202431s) [noise=true , address=45.91.116.12:4225 , piece_id=LEUVZRRXNMITUVUNGPCGIFJLITDNNHHU2LLITK5HGL43TGWH3Y6A, seg_pos={0 0}, upload=451246]: Done uploading piece. err: <nil> Dec 27 01:43:21.124139864 (514h14m20.995222899s) [seg_pos={0 0}, upload=451246]: Segment reached optimal threshold of 13 pieces. Dec 27 01:43:21.124394350 (514h14m20.995477395s) [noise=true , address=45.91.116.6:4974 , piece_id=E63LNYCHIBDF4UTBF4VXOAEBVGDXQP36Z77PLHFBAOOHIW2LAEMA, seg_pos={0 0}, upload=451246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 01:43:21.124402585 (514h14m20.99548564s) [noise=true , address=45.91.116.6:4974 , piece_id=E63LNYCHIBDF4UTBF4VXOAEBVGDXQP36Z77PLHFBAOOHIW2LAEMA, seg_pos={0 0}, upload=451246]: Long tail cancellation due to optimal threshold Dec 27 01:43:21.124437662 (514h14m20.995520707s) [noise=true , address=45.91.116.8:4045 , piece_id=BENM567HPTFMNSFUP2PH6JN56FD2OH6B7KTCBWJT665DH4NSKALQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 01:43:21.124445547 (514h14m20.995528582s) [noise=true , address=45.91.116.8:4045 , piece_id=BENM567HPTFMNSFUP2PH6JN56FD2OH6B7KTCBWJT665DH4NSKALQ, seg_pos={0 0}, upload=451246]: Long tail cancellation due to optimal threshold Dec 27 01:43:21.124479321 (514h14m20.995562366s) [noise=true , address=77.90.31.10:4519 , piece_id=USHS6FX4NDTQ7Q6IBV73K64YE2V3TFGPFM54VGOI6SWJT66WMUPQ, seg_pos={0 0}, upload=451246]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 01:43:21.124486224 (514h14m20.995569269s) [noise=true , address=77.90.31.10:4519 , piece_id=USHS6FX4NDTQ7Q6IBV73K64YE2V3TFGPFM54VGOI6SWJT66WMUPQ, seg_pos={0 0}, upload=451246]: Long tail cancellation due to optimal threshold Dec 27 01:43:21.124502986 (514h14m20.995586021s) [seg_pos={0 0}, upload=451246]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:43:21.124594320 (514h14m20.995677365s) [upload=451246]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:43:21.124603388 (514h14m20.995686423s) [upload=451246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:43:21.407069742 (514h14m21.278152787s) [upload=451246]: Done uploading segments... Dec 27 01:43:21.407091623 (514h14m21.278174668s) [upload=451246]: Upload finished. err: <nil> Duration: 30.003099974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X.] duration: 6.467823452s trace: https://jaeger.dev.storj.io/trace/1eef4eef40dacd7e FAIL (2025-12-27 01:42:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ) context deadline exceeded ======LOGS====== gments... Dec 27 01:41:23.521759923 (514h12m23.392842968s) [upload=451219]: Got next segment. Inline: false Dec 27 01:41:23.521864773 (514h12m23.392947808s) [upload=451219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:41:23.521872888 (514h12m23.392955933s) [upload=451219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:41:23.767709064 (514h12m23.638792099s) [seg_pos={0 0}, upload=451219]: Begin upload segment... Dec 27 01:41:23.767769359 (514h12m23.638852394s) [seg_pos={0 0}, upload=451219]: Done begin upload segment. Dec 27 01:41:23.767798675 (514h12m23.63888171s) [noise=true , address=77.90.31.21:4880 , piece_id=KIIPHJMOWX3G6SS2FJQR6ECWRVWRU7NZFL64AMTKPGR2OZ6ERVJA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.767843790 (514h12m23.638926825s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.768422421 (514h12m23.639505456s) [noise=true , address=45.91.116.14:4192 , piece_id=OOFVLYF4ZMI3G5ZSGSMAGEIIBCTETXBROUWO5G5NQHQFKBZ37I7Q, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.768957791 (514h12m23.640040837s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.769584194 (514h12m23.640667229s) [noise=true , address=145.79.227.48:4302 , piece_id=N3BAASULRV7NJ7PO4U57OJM3MMUHJZ3DL5IF4UKD3UUCFPBVY44A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.770090628 (514h12m23.641173683s) [noise=true , address=45.91.116.9:4264 , piece_id=Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.770634765 (514h12m23.64171781s) [noise=true , address=45.91.116.8:4045 , piece_id=IT7ZAUFBMCYJ5GS4YTIF7Q25JM5ZR2L3KRJAR2GQ4LYPIGWSNHXA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.771107103 (514h12m23.642190148s) [noise=true , address=145.79.227.5:4222 , piece_id=3RA3UF455GNELDVRWCTATB3PJSNPKLEPFDBKWLCGPGNHZECQMQSQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.771593940 (514h12m23.642676975s) [noise=true , address=45.91.116.6:4973 , piece_id=HHO4EXWCSHHUFPYXL27GWWW7W7PJWKZZXZQXGBDE3KYFME3IEEKA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.772075047 (514h12m23.643158082s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.772554239 (514h12m23.643637284s) [noise=true , address=145.79.227.41:4087 , piece_id=O4HGP3EAOI6N4NCMBGW3PW7NRWX3CMHKIF7FJUMKLW442ZM2DWIQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.773045445 (514h12m23.64412849s) [noise=true , address=145.79.227.23:4545 , piece_id=HPIVO4D2XN4CJ26AZZWOFSLDIDVLTQRBAOYGOYVLEKREJMJ3XUHA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774030741 (514h12m23.645113776s) [noise=true , address=145.79.227.39:4013 , piece_id=5QYHZ6PUCPCZCV7Z4KEPIM2PXQ5CZ7S4SP3KBVUJJMDXPCR7WPOQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774519732 (514h12m23.645602767s) [noise=true , address=45.91.116.5:4298 , piece_id=3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774928701 (514h12m23.646011736s) [noise=true , address=145.79.227.37:4565 , piece_id=4QGNF2X4JMLXOFHQKSA2FVJAQ2ORHPUUVP42KY75XXA6M32CKGYA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.775306500 (514h12m23.646389535s) [noise=true , address=145.79.227.38:4977 , piece_id=GG7T6OYK3T6XHR7X4TGSYVYVNA2I7MPYADWNKMA5BYEHIQKUQX7A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:25.462744065 (514h12m25.33382711s) [upload=451219]: Next returned nil segment Dec 27 01:41:25.462762079 (514h12m25.333845134s) [upload=451219]: Waiting for error group managing segments... Dec 27 01:41:25.762750378 (514h12m25.633833423s) [noise=true , address=145.79.227.39:4013 , piece_id=5QYHZ6PUCPCZCV7Z4KEPIM2PXQ5CZ7S4SP3KBVUJJMDXPCR7WPOQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.781206701 (514h12m25.652289747s) [noise=true , address=145.79.227.48:4302 , piece_id=N3BAASULRV7NJ7PO4U57OJM3MMUHJZ3DL5IF4UKD3UUCFPBVY44A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.792264407 (514h12m25.663347442s) [noise=true , address=145.79.227.38:4977 , piece_id=GG7T6OYK3T6XHR7X4TGSYVYVNA2I7MPYADWNKMA5BYEHIQKUQX7A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.796099489 (514h12m25.667182524s) [noise=true , address=145.79.227.5:4222 , piece_id=3RA3UF455GNELDVRWCTATB3PJSNPKLEPFDBKWLCGPGNHZECQMQSQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.799594785 (514h12m25.67067782s) [noise=true , address=145.79.227.41:4087 , piece_id=O4HGP3EAOI6N4NCMBGW3PW7NRWX3CMHKIF7FJUMKLW442ZM2DWIQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.800360263 (514h12m25.671443308s) [noise=true , address=145.79.227.23:4545 , piece_id=HPIVO4D2XN4CJ26AZZWOFSLDIDVLTQRBAOYGOYVLEKREJMJ3XUHA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.801710714 (514h12m25.672793749s) [noise=true , address=145.79.227.37:4565 , piece_id=4QGNF2X4JMLXOFHQKSA2FVJAQ2ORHPUUVP42KY75XXA6M32CKGYA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.000104273 (514h12m25.871187338s) [noise=true , address=45.91.116.14:4192 , piece_id=OOFVLYF4ZMI3G5ZSGSMAGEIIBCTETXBROUWO5G5NQHQFKBZ37I7Q, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.012016625 (514h12m25.88309967s) [noise=true , address=77.90.31.21:4880 , piece_id=KIIPHJMOWX3G6SS2FJQR6ECWRVWRU7NZFL64AMTKPGR2OZ6ERVJA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.023649516 (514h12m25.894732561s) [noise=true , address=45.91.116.6:4973 , piece_id=HHO4EXWCSHHUFPYXL27GWWW7W7PJWKZZXZQXGBDE3KYFME3IEEKA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.086912698 (514h12m25.957995733s) [noise=true , address=45.91.116.5:4298 , piece_id=3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.115096173 (514h12m25.986179208s) [noise=true , address=45.91.116.9:4264 , piece_id=Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.149279002 (514h12m26.020362037s) [noise=true , address=45.91.116.8:4045 , piece_id=IT7ZAUFBMCYJ5GS4YTIF7Q25JM5ZR2L3KRJAR2GQ4LYPIGWSNHXA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.149300432 (514h12m26.020383467s) [seg_pos={0 0}, upload=451219]: Segment reached optimal threshold of 13 pieces. Dec 27 01:41:26.149465076 (514h12m26.020548121s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection Dec 27 01:41:26.149472881 (514h12m26.020555926s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149784624 (514h12m26.02086766s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 01:41:26.149792681 (514h12m26.020875726s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149937115 (514h12m26.021020161s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection Dec 27 01:41:26.149947114 (514h12m26.021030159s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149971591 (514h12m26.021054636s) [seg_pos={0 0}, upload=451219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:41:26.150050812 (514h12m26.021133857s) [upload=451219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:41:26.150057504 (514h12m26.021140549s) [upload=451219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:41:26.441327312 (514h12m26.312410357s) [upload=451219]: Done uploading segments... Dec 27 01:41:26.441348973 (514h12m26.312432018s) [upload=451219]: Upload finished. err: <nil> Duration: 6.46784961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..X] duration: 30.003216333s trace: https://jaeger.dev.storj.io/trace/1eef4eef40dacd7e FAIL (2025-12-27 01:41:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ) context deadline exceeded ======LOGS====== gments... Dec 27 01:41:23.521759923 (514h12m23.392842968s) [upload=451219]: Got next segment. Inline: false Dec 27 01:41:23.521864773 (514h12m23.392947808s) [upload=451219]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:41:23.521872888 (514h12m23.392955933s) [upload=451219]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:41:23.767709064 (514h12m23.638792099s) [seg_pos={0 0}, upload=451219]: Begin upload segment... Dec 27 01:41:23.767769359 (514h12m23.638852394s) [seg_pos={0 0}, upload=451219]: Done begin upload segment. Dec 27 01:41:23.767798675 (514h12m23.63888171s) [noise=true , address=77.90.31.21:4880 , piece_id=KIIPHJMOWX3G6SS2FJQR6ECWRVWRU7NZFL64AMTKPGR2OZ6ERVJA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.767843790 (514h12m23.638926825s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.768422421 (514h12m23.639505456s) [noise=true , address=45.91.116.14:4192 , piece_id=OOFVLYF4ZMI3G5ZSGSMAGEIIBCTETXBROUWO5G5NQHQFKBZ37I7Q, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.768957791 (514h12m23.640040837s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.769584194 (514h12m23.640667229s) [noise=true , address=145.79.227.48:4302 , piece_id=N3BAASULRV7NJ7PO4U57OJM3MMUHJZ3DL5IF4UKD3UUCFPBVY44A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.770090628 (514h12m23.641173683s) [noise=true , address=45.91.116.9:4264 , piece_id=Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.770634765 (514h12m23.64171781s) [noise=true , address=45.91.116.8:4045 , piece_id=IT7ZAUFBMCYJ5GS4YTIF7Q25JM5ZR2L3KRJAR2GQ4LYPIGWSNHXA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.771107103 (514h12m23.642190148s) [noise=true , address=145.79.227.5:4222 , piece_id=3RA3UF455GNELDVRWCTATB3PJSNPKLEPFDBKWLCGPGNHZECQMQSQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.771593940 (514h12m23.642676975s) [noise=true , address=45.91.116.6:4973 , piece_id=HHO4EXWCSHHUFPYXL27GWWW7W7PJWKZZXZQXGBDE3KYFME3IEEKA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.772075047 (514h12m23.643158082s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.772554239 (514h12m23.643637284s) [noise=true , address=145.79.227.41:4087 , piece_id=O4HGP3EAOI6N4NCMBGW3PW7NRWX3CMHKIF7FJUMKLW442ZM2DWIQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.773045445 (514h12m23.64412849s) [noise=true , address=145.79.227.23:4545 , piece_id=HPIVO4D2XN4CJ26AZZWOFSLDIDVLTQRBAOYGOYVLEKREJMJ3XUHA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774030741 (514h12m23.645113776s) [noise=true , address=145.79.227.39:4013 , piece_id=5QYHZ6PUCPCZCV7Z4KEPIM2PXQ5CZ7S4SP3KBVUJJMDXPCR7WPOQ, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774519732 (514h12m23.645602767s) [noise=true , address=45.91.116.5:4298 , piece_id=3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.774928701 (514h12m23.646011736s) [noise=true , address=145.79.227.37:4565 , piece_id=4QGNF2X4JMLXOFHQKSA2FVJAQ2ORHPUUVP42KY75XXA6M32CKGYA, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:23.775306500 (514h12m23.646389535s) [noise=true , address=145.79.227.38:4977 , piece_id=GG7T6OYK3T6XHR7X4TGSYVYVNA2I7MPYADWNKMA5BYEHIQKUQX7A, seg_pos={0 0}, upload=451219]: Uploading piece... Dec 27 01:41:25.462744065 (514h12m25.33382711s) [upload=451219]: Next returned nil segment Dec 27 01:41:25.462762079 (514h12m25.333845134s) [upload=451219]: Waiting for error group managing segments... Dec 27 01:41:25.762750378 (514h12m25.633833423s) [noise=true , address=145.79.227.39:4013 , piece_id=5QYHZ6PUCPCZCV7Z4KEPIM2PXQ5CZ7S4SP3KBVUJJMDXPCR7WPOQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.781206701 (514h12m25.652289747s) [noise=true , address=145.79.227.48:4302 , piece_id=N3BAASULRV7NJ7PO4U57OJM3MMUHJZ3DL5IF4UKD3UUCFPBVY44A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.792264407 (514h12m25.663347442s) [noise=true , address=145.79.227.38:4977 , piece_id=GG7T6OYK3T6XHR7X4TGSYVYVNA2I7MPYADWNKMA5BYEHIQKUQX7A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.796099489 (514h12m25.667182524s) [noise=true , address=145.79.227.5:4222 , piece_id=3RA3UF455GNELDVRWCTATB3PJSNPKLEPFDBKWLCGPGNHZECQMQSQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.799594785 (514h12m25.67067782s) [noise=true , address=145.79.227.41:4087 , piece_id=O4HGP3EAOI6N4NCMBGW3PW7NRWX3CMHKIF7FJUMKLW442ZM2DWIQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.800360263 (514h12m25.671443308s) [noise=true , address=145.79.227.23:4545 , piece_id=HPIVO4D2XN4CJ26AZZWOFSLDIDVLTQRBAOYGOYVLEKREJMJ3XUHA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:25.801710714 (514h12m25.672793749s) [noise=true , address=145.79.227.37:4565 , piece_id=4QGNF2X4JMLXOFHQKSA2FVJAQ2ORHPUUVP42KY75XXA6M32CKGYA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.000104273 (514h12m25.871187338s) [noise=true , address=45.91.116.14:4192 , piece_id=OOFVLYF4ZMI3G5ZSGSMAGEIIBCTETXBROUWO5G5NQHQFKBZ37I7Q, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.012016625 (514h12m25.88309967s) [noise=true , address=77.90.31.21:4880 , piece_id=KIIPHJMOWX3G6SS2FJQR6ECWRVWRU7NZFL64AMTKPGR2OZ6ERVJA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.023649516 (514h12m25.894732561s) [noise=true , address=45.91.116.6:4973 , piece_id=HHO4EXWCSHHUFPYXL27GWWW7W7PJWKZZXZQXGBDE3KYFME3IEEKA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.086912698 (514h12m25.957995733s) [noise=true , address=45.91.116.5:4298 , piece_id=3FOXQMW5FZBOV2ODEVZE7WFIIQMXSBSX7CIP5ARC5XABFNEN2MVA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.115096173 (514h12m25.986179208s) [noise=true , address=45.91.116.9:4264 , piece_id=Z5XMF7YTFIWRDSX4TAPLOVGK7OEZCJSWWJSURJVQM6WJA6M5YJZQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.149279002 (514h12m26.020362037s) [noise=true , address=45.91.116.8:4045 , piece_id=IT7ZAUFBMCYJ5GS4YTIF7Q25JM5ZR2L3KRJAR2GQ4LYPIGWSNHXA, seg_pos={0 0}, upload=451219]: Done uploading piece. err: <nil> Dec 27 01:41:26.149300432 (514h12m26.020383467s) [seg_pos={0 0}, upload=451219]: Segment reached optimal threshold of 13 pieces. Dec 27 01:41:26.149465076 (514h12m26.020548121s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40712->145.79.227.6:4259: use of closed network connection Dec 27 01:41:26.149472881 (514h12m26.020555926s) [noise=true , address=145.79.227.6:4259 , piece_id=NKGAER6DHCM4Z4ADY3F4ZCI4MBU5TFNEZ6ONGWWD2WXFOSVZF32A, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149784624 (514h12m26.02086766s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 01:41:26.149792681 (514h12m26.020875726s) [noise=true , address=45.91.116.4:4937 , piece_id=54LGO7GSSDG5CM3DWICSLIXLVFAPS6Q57JUTIGMEU32P7JIUMX4A, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149937115 (514h12m26.021020161s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47948->45.91.116.11:4117: use of closed network connection Dec 27 01:41:26.149947114 (514h12m26.021030159s) [noise=true , address=45.91.116.11:4117 , piece_id=XJF4TWI34WF7SNF43D3GXTCZA3OGCMAP72N5PRCR36GIMX32PBSQ, seg_pos={0 0}, upload=451219]: Long tail cancellation due to optimal threshold Dec 27 01:41:26.149971591 (514h12m26.021054636s) [seg_pos={0 0}, upload=451219]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:41:26.150050812 (514h12m26.021133857s) [upload=451219]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:41:26.150057504 (514h12m26.021140549s) [upload=451219]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:41:26.441327312 (514h12m26.312410357s) [upload=451219]: Done uploading segments... Dec 27 01:41:26.441348973 (514h12m26.312432018s) [upload=451219]: Upload finished. err: <nil> Duration: 30.00322951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X..] duration: 6.694352647s trace: https://jaeger.dev.storj.io/trace/3131f75519464479 FAIL (2025-12-27 01:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA) context deadline exceeded ======LOGS====== ]: Starting upload Dec 27 01:39:14.829755941 (514h10m14.700838976s) [upload=451187]: Done starting upload Dec 27 01:39:14.829917649 (514h10m14.701000704s) [upload=451187]: Uploading segments... Dec 27 01:39:14.830085589 (514h10m14.701168624s) [upload=451187]: Got next segment. Inline: false Dec 27 01:39:14.830141274 (514h10m14.70122431s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:39:14.830146464 (514h10m14.701229499s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:39:15.074706791 (514h10m14.945789836s) [seg_pos={0 0}, upload=451187]: Begin upload segment... Dec 27 01:39:15.074748842 (514h10m14.945831877s) [seg_pos={0 0}, upload=451187]: Done begin upload segment. Dec 27 01:39:15.074782265 (514h10m14.94586531s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.074853892 (514h10m14.945936937s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075411383 (514h10m14.946494418s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075489061 (514h10m14.946572096s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075878542 (514h10m14.946961588s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075941202 (514h10m14.947024237s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076255641 (514h10m14.947338686s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076420725 (514h10m14.94750377s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076586300 (514h10m14.947669345s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076864520 (514h10m14.947947555s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076928682 (514h10m14.948011727s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077116640 (514h10m14.948199685s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077369401 (514h10m14.948452447s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077415449 (514h10m14.948498484s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077983240 (514h10m14.949066285s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.078555630 (514h10m14.949638675s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:16.787250532 (514h10m16.658333567s) [upload=451187]: Next returned nil segment Dec 27 01:39:16.787267634 (514h10m16.658350679s) [upload=451187]: Waiting for error group managing segments... Dec 27 01:39:17.067473726 (514h10m16.938556761s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.077397262 (514h10m16.948480307s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.082072023 (514h10m16.953155058s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.095112313 (514h10m16.966195358s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.096375488 (514h10m16.967458533s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.100724630 (514h10m16.971807675s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105263913 (514h10m16.976346958s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105693260 (514h10m16.976776305s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.108925545 (514h10m16.98000858s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.215578898 (514h10m17.086661933s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.265018087 (514h10m17.136101122s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.306784526 (514h10m17.177867561s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339425489 (514h10m17.210508524s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339443824 (514h10m17.210526869s) [seg_pos={0 0}, upload=451187]: Segment reached optimal threshold of 13 pieces. Dec 27 01:39:17.339822154 (514h10m17.210905189s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 01:39:17.339830381 (514h10m17.210913416s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340105564 (514h10m17.211188599s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection Dec 27 01:39:17.340113019 (514h10m17.211196064s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340452875 (514h10m17.21153591s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:54114->145.79.227.3:4150: use of closed network connection; context canceled Dec 27 01:39:17.340460631 (514h10m17.211543666s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340475859 (514h10m17.211558894s) [seg_pos={0 0}, upload=451187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:39:17.340560200 (514h10m17.211643246s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:39:17.340565901 (514h10m17.211648946s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:39:17.633476844 (514h10m17.504559879s) [upload=451187]: Done uploading segments... Dec 27 01:39:17.633495520 (514h10m17.504578565s) [upload=451187]: Upload finished. err: <nil> Duration: 6.694379157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X.] duration: 5.932791412s trace: https://jaeger.dev.storj.io/trace/3131f75519464479 FAIL (2025-12-27 01:40:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA) context deadline exceeded ======LOGS====== ]: Starting upload Dec 27 01:39:14.829755941 (514h10m14.700838976s) [upload=451187]: Done starting upload Dec 27 01:39:14.829917649 (514h10m14.701000704s) [upload=451187]: Uploading segments... Dec 27 01:39:14.830085589 (514h10m14.701168624s) [upload=451187]: Got next segment. Inline: false Dec 27 01:39:14.830141274 (514h10m14.70122431s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:39:14.830146464 (514h10m14.701229499s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:39:15.074706791 (514h10m14.945789836s) [seg_pos={0 0}, upload=451187]: Begin upload segment... Dec 27 01:39:15.074748842 (514h10m14.945831877s) [seg_pos={0 0}, upload=451187]: Done begin upload segment. Dec 27 01:39:15.074782265 (514h10m14.94586531s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.074853892 (514h10m14.945936937s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075411383 (514h10m14.946494418s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075489061 (514h10m14.946572096s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075878542 (514h10m14.946961588s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075941202 (514h10m14.947024237s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076255641 (514h10m14.947338686s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076420725 (514h10m14.94750377s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076586300 (514h10m14.947669345s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076864520 (514h10m14.947947555s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076928682 (514h10m14.948011727s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077116640 (514h10m14.948199685s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077369401 (514h10m14.948452447s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077415449 (514h10m14.948498484s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077983240 (514h10m14.949066285s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.078555630 (514h10m14.949638675s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:16.787250532 (514h10m16.658333567s) [upload=451187]: Next returned nil segment Dec 27 01:39:16.787267634 (514h10m16.658350679s) [upload=451187]: Waiting for error group managing segments... Dec 27 01:39:17.067473726 (514h10m16.938556761s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.077397262 (514h10m16.948480307s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.082072023 (514h10m16.953155058s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.095112313 (514h10m16.966195358s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.096375488 (514h10m16.967458533s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.100724630 (514h10m16.971807675s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105263913 (514h10m16.976346958s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105693260 (514h10m16.976776305s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.108925545 (514h10m16.98000858s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.215578898 (514h10m17.086661933s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.265018087 (514h10m17.136101122s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.306784526 (514h10m17.177867561s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339425489 (514h10m17.210508524s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339443824 (514h10m17.210526869s) [seg_pos={0 0}, upload=451187]: Segment reached optimal threshold of 13 pieces. Dec 27 01:39:17.339822154 (514h10m17.210905189s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 01:39:17.339830381 (514h10m17.210913416s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340105564 (514h10m17.211188599s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection Dec 27 01:39:17.340113019 (514h10m17.211196064s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340452875 (514h10m17.21153591s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:54114->145.79.227.3:4150: use of closed network connection; context canceled Dec 27 01:39:17.340460631 (514h10m17.211543666s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340475859 (514h10m17.211558894s) [seg_pos={0 0}, upload=451187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:39:17.340560200 (514h10m17.211643246s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:39:17.340565901 (514h10m17.211648946s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:39:17.633476844 (514h10m17.504559879s) [upload=451187]: Done uploading segments... Dec 27 01:39:17.633495520 (514h10m17.504578565s) [upload=451187]: Upload finished. err: <nil> Duration: 5.932820988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..X] duration: 30.0027629s trace: https://jaeger.dev.storj.io/trace/3131f75519464479 FAIL (2025-12-27 01:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA) context deadline exceeded ======LOGS====== ]: Starting upload Dec 27 01:39:14.829755941 (514h10m14.700838976s) [upload=451187]: Done starting upload Dec 27 01:39:14.829917649 (514h10m14.701000704s) [upload=451187]: Uploading segments... Dec 27 01:39:14.830085589 (514h10m14.701168624s) [upload=451187]: Got next segment. Inline: false Dec 27 01:39:14.830141274 (514h10m14.70122431s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:39:14.830146464 (514h10m14.701229499s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:39:15.074706791 (514h10m14.945789836s) [seg_pos={0 0}, upload=451187]: Begin upload segment... Dec 27 01:39:15.074748842 (514h10m14.945831877s) [seg_pos={0 0}, upload=451187]: Done begin upload segment. Dec 27 01:39:15.074782265 (514h10m14.94586531s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.074853892 (514h10m14.945936937s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075411383 (514h10m14.946494418s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075489061 (514h10m14.946572096s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075878542 (514h10m14.946961588s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.075941202 (514h10m14.947024237s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076255641 (514h10m14.947338686s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076420725 (514h10m14.94750377s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076586300 (514h10m14.947669345s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076864520 (514h10m14.947947555s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.076928682 (514h10m14.948011727s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077116640 (514h10m14.948199685s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077369401 (514h10m14.948452447s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077415449 (514h10m14.948498484s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.077983240 (514h10m14.949066285s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:15.078555630 (514h10m14.949638675s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Uploading piece... Dec 27 01:39:16.787250532 (514h10m16.658333567s) [upload=451187]: Next returned nil segment Dec 27 01:39:16.787267634 (514h10m16.658350679s) [upload=451187]: Waiting for error group managing segments... Dec 27 01:39:17.067473726 (514h10m16.938556761s) [noise=true , address=145.79.227.38:4977 , piece_id=RLS4W5AMRCWXQCKMM6LGI3LU2ZVBCKOMZ6YO5GJB643C5MPSQVWA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.077397262 (514h10m16.948480307s) [noise=true , address=145.79.227.37:4565 , piece_id=KWSFCORESNYF4QNRI2JNIYZY4A6GM5ATVL3ANBWRHWNCELZA73SA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.082072023 (514h10m16.953155058s) [noise=true , address=145.79.227.11:4732 , piece_id=3LDRYJIDKGKBLQKS63YCLDH6WPVGXPDMX5IXQXNLMMFE7B6CM2ZQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.095112313 (514h10m16.966195358s) [noise=true , address=145.79.227.17:4972 , piece_id=BHDFUGDNMY7R5RNZHDJBGELS7N3RSJIELSCNPVIJEG7MSBAVW3FA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.096375488 (514h10m16.967458533s) [noise=true , address=145.79.227.45:4230 , piece_id=V6WSSIQBXN7PXFJSFLWNRCAZG32Z3EBEGAK6SZNF5524TAET6VOQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.100724630 (514h10m16.971807675s) [noise=true , address=145.79.227.50:4373 , piece_id=HC3OXQ5RGX7IPNB62FJCKXMW4LO7MULAMFJT6RO7UD2F4LSZDJTQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105263913 (514h10m16.976346958s) [noise=true , address=145.79.227.16:4909 , piece_id=3FE7L3SXXW524RDRAOWFOXZVMO2ZEVTKG2565HZZOZ5GESYCWBLQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.105693260 (514h10m16.976776305s) [noise=true , address=145.79.227.52:4447 , piece_id=6DFIOFXBB4MGDZ3NBANWYQHPQGFU2PLFSPIJKGIRQEIGESKQBF6Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.108925545 (514h10m16.98000858s) [noise=true , address=145.79.227.48:4303 , piece_id=SRVX5NX2C3B33BNYFCXP2Z475ZNT5Y3NSAQJ62PYPYICLOW5574Q, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.215578898 (514h10m17.086661933s) [noise=true , address=145.79.227.29:4942 , piece_id=ISXKY7NRNIGGBR63U5THYEZX4A2M6T2GTJEOGEZ2WPN5ZWM75XFA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.265018087 (514h10m17.136101122s) [noise=true , address=77.90.31.19:4841 , piece_id=ZILVKQRJPD6VY4BV2VRNRBBNJIOBCCJHNISWVNQ4BUOV62SBYWIA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.306784526 (514h10m17.177867561s) [noise=true , address=45.91.116.13:4153 , piece_id=RZE3QOAEWRSFLOJ2IQ22RSJ6NESHI4HMYZ45PF7S4DMQBLYMPBMA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339425489 (514h10m17.210508524s) [noise=true , address=45.91.116.14:4189 , piece_id=BE2T5RFGJZ7LKDMLKGBWBJH6CSOPU65UANAGYG6DWO3K5LB5RD2A, seg_pos={0 0}, upload=451187]: Done uploading piece. err: <nil> Dec 27 01:39:17.339443824 (514h10m17.210526869s) [seg_pos={0 0}, upload=451187]: Segment reached optimal threshold of 13 pieces. Dec 27 01:39:17.339822154 (514h10m17.210905189s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 01:39:17.339830381 (514h10m17.210913416s) [noise=true , address=77.90.31.14:4661 , piece_id=5C2UCAIEUXKAU5UE4XWROBIBFVLJWJKUTG2H6Z55UZSVEIFHVISA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340105564 (514h10m17.211188599s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42228->45.91.116.9:4261: use of closed network connection Dec 27 01:39:17.340113019 (514h10m17.211196064s) [noise=true , address=45.91.116.9:4261 , piece_id=3FGGWESFWZXGNS62LEDUH57SH5EHGXRGIOFQLP4UXIFC5E43DKFQ, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340452875 (514h10m17.21153591s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:54114->145.79.227.3:4150: use of closed network connection; context canceled Dec 27 01:39:17.340460631 (514h10m17.211543666s) [noise=true , address=145.79.227.3:4150 , piece_id=5M4ARYPZ2AHM2UIGID6KLAREK7272KWC6ZSF3N5NCEYRCIENI6AA, seg_pos={0 0}, upload=451187]: Long tail cancellation due to optimal threshold Dec 27 01:39:17.340475859 (514h10m17.211558894s) [seg_pos={0 0}, upload=451187]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:39:17.340560200 (514h10m17.211643246s) [upload=451187]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:39:17.340565901 (514h10m17.211648946s) [upload=451187]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:39:17.633476844 (514h10m17.504559879s) [upload=451187]: Done uploading segments... Dec 27 01:39:17.633495520 (514h10m17.504578565s) [upload=451187]: Upload finished. err: <nil> Duration: 30.002797164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX..] duration: 28.485739777s trace: https://jaeger.dev.storj.io/trace/558e5eccd38ab503 FAIL (2025-12-27 01:38:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA) context deadline exceeded ======LOGS====== m41.13155324s) [upload=451146]: Uploading segments... Dec 27 01:36:41.260512926 (514h7m41.131595961s) [upload=451146]: Got next segment. Inline: false Dec 27 01:36:41.260540178 (514h7m41.131623213s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:36:41.260544195 (514h7m41.13162723s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:36:41.508758190 (514h7m41.379841225s) [seg_pos={0 0}, upload=451146]: Begin upload segment... Dec 27 01:36:41.508802144 (514h7m41.379885179s) [seg_pos={0 0}, upload=451146]: Done begin upload segment. Dec 27 01:36:41.508830929 (514h7m41.379913974s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.508876124 (514h7m41.379959159s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509124417 (514h7m41.380207452s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509437143 (514h7m41.380520178s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509631974 (514h7m41.380715019s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509705073 (514h7m41.380788108s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510024180 (514h7m41.381107216s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510097340 (514h7m41.381180375s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510531256 (514h7m41.381614292s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510640955 (514h7m41.38172399s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510976705 (514h7m41.38205974s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511424296 (514h7m41.382507341s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511814680 (514h7m41.382897716s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512200806 (514h7m41.383283841s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512626366 (514h7m41.383709411s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512959891 (514h7m41.384042926s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:43.227976839 (514h7m43.099059874s) [upload=451146]: Next returned nil segment Dec 27 01:36:43.228025171 (514h7m43.099108206s) [upload=451146]: Waiting for error group managing segments... Dec 27 01:36:43.518576881 (514h7m43.389659917s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.525714112 (514h7m43.396797147s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.531477747 (514h7m43.402560792s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541361136 (514h7m43.412444181s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541452310 (514h7m43.412535355s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.547434571 (514h7m43.418517606s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.761126180 (514h7m43.632209215s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792487124 (514h7m43.663570159s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792976496 (514h7m43.664059541s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.806870573 (514h7m43.677953608s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.824405031 (514h7m43.695488066s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.856319741 (514h7m43.727402786s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879719408 (514h7m43.750802453s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879739525 (514h7m43.750822561s) [seg_pos={0 0}, upload=451146]: Segment reached optimal threshold of 13 pieces. Dec 27 01:36:43.879906464 (514h7m43.750989499s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection Dec 27 01:36:43.879912375 (514h7m43.75099542s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880028446 (514h7m43.751111491s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 01:36:43.880046109 (514h7m43.751129145s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880063112 (514h7m43.751146147s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection Dec 27 01:36:43.880072330 (514h7m43.751155365s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880121062 (514h7m43.751204097s) [seg_pos={0 0}, upload=451146]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:36:43.880230500 (514h7m43.751313535s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:36:43.880239648 (514h7m43.751322724s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:36:44.172297988 (514h7m44.043381023s) [upload=451146]: Done uploading segments... Dec 27 01:36:44.172314539 (514h7m44.043397584s) [upload=451146]: Upload finished. err: <nil> Duration: 28.485768041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX.] duration: 6.026588931s trace: https://jaeger.dev.storj.io/trace/558e5eccd38ab503 FAIL (2025-12-27 01:37:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA) context deadline exceeded ======LOGS====== m41.13155324s) [upload=451146]: Uploading segments... Dec 27 01:36:41.260512926 (514h7m41.131595961s) [upload=451146]: Got next segment. Inline: false Dec 27 01:36:41.260540178 (514h7m41.131623213s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:36:41.260544195 (514h7m41.13162723s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:36:41.508758190 (514h7m41.379841225s) [seg_pos={0 0}, upload=451146]: Begin upload segment... Dec 27 01:36:41.508802144 (514h7m41.379885179s) [seg_pos={0 0}, upload=451146]: Done begin upload segment. Dec 27 01:36:41.508830929 (514h7m41.379913974s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.508876124 (514h7m41.379959159s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509124417 (514h7m41.380207452s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509437143 (514h7m41.380520178s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509631974 (514h7m41.380715019s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509705073 (514h7m41.380788108s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510024180 (514h7m41.381107216s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510097340 (514h7m41.381180375s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510531256 (514h7m41.381614292s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510640955 (514h7m41.38172399s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510976705 (514h7m41.38205974s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511424296 (514h7m41.382507341s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511814680 (514h7m41.382897716s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512200806 (514h7m41.383283841s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512626366 (514h7m41.383709411s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512959891 (514h7m41.384042926s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:43.227976839 (514h7m43.099059874s) [upload=451146]: Next returned nil segment Dec 27 01:36:43.228025171 (514h7m43.099108206s) [upload=451146]: Waiting for error group managing segments... Dec 27 01:36:43.518576881 (514h7m43.389659917s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.525714112 (514h7m43.396797147s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.531477747 (514h7m43.402560792s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541361136 (514h7m43.412444181s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541452310 (514h7m43.412535355s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.547434571 (514h7m43.418517606s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.761126180 (514h7m43.632209215s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792487124 (514h7m43.663570159s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792976496 (514h7m43.664059541s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.806870573 (514h7m43.677953608s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.824405031 (514h7m43.695488066s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.856319741 (514h7m43.727402786s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879719408 (514h7m43.750802453s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879739525 (514h7m43.750822561s) [seg_pos={0 0}, upload=451146]: Segment reached optimal threshold of 13 pieces. Dec 27 01:36:43.879906464 (514h7m43.750989499s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection Dec 27 01:36:43.879912375 (514h7m43.75099542s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880028446 (514h7m43.751111491s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 01:36:43.880046109 (514h7m43.751129145s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880063112 (514h7m43.751146147s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection Dec 27 01:36:43.880072330 (514h7m43.751155365s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880121062 (514h7m43.751204097s) [seg_pos={0 0}, upload=451146]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:36:43.880230500 (514h7m43.751313535s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:36:43.880239648 (514h7m43.751322724s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:36:44.172297988 (514h7m44.043381023s) [upload=451146]: Done uploading segments... Dec 27 01:36:44.172314539 (514h7m44.043397584s) [upload=451146]: Upload finished. err: <nil> Duration: 6.026616522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XXX] duration: 30.004245629s trace: https://jaeger.dev.storj.io/trace/558e5eccd38ab503 FAIL (2025-12-27 01:37:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA) context deadline exceeded ======LOGS====== m41.13155324s) [upload=451146]: Uploading segments... Dec 27 01:36:41.260512926 (514h7m41.131595961s) [upload=451146]: Got next segment. Inline: false Dec 27 01:36:41.260540178 (514h7m41.131623213s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:36:41.260544195 (514h7m41.13162723s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:36:41.508758190 (514h7m41.379841225s) [seg_pos={0 0}, upload=451146]: Begin upload segment... Dec 27 01:36:41.508802144 (514h7m41.379885179s) [seg_pos={0 0}, upload=451146]: Done begin upload segment. Dec 27 01:36:41.508830929 (514h7m41.379913974s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.508876124 (514h7m41.379959159s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509124417 (514h7m41.380207452s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509437143 (514h7m41.380520178s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509631974 (514h7m41.380715019s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.509705073 (514h7m41.380788108s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510024180 (514h7m41.381107216s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510097340 (514h7m41.381180375s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510531256 (514h7m41.381614292s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510640955 (514h7m41.38172399s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.510976705 (514h7m41.38205974s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511424296 (514h7m41.382507341s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.511814680 (514h7m41.382897716s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512200806 (514h7m41.383283841s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512626366 (514h7m41.383709411s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:41.512959891 (514h7m41.384042926s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Uploading piece... Dec 27 01:36:43.227976839 (514h7m43.099059874s) [upload=451146]: Next returned nil segment Dec 27 01:36:43.228025171 (514h7m43.099108206s) [upload=451146]: Waiting for error group managing segments... Dec 27 01:36:43.518576881 (514h7m43.389659917s) [noise=true , address=145.79.227.41:4085 , piece_id=TD2RHYZ7UCCQ2FU2HGQ2HOOCLN4FRIUHQOMFQTO4SGONWC3JNPHQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.525714112 (514h7m43.396797147s) [noise=true , address=145.79.227.11:4731 , piece_id=CYVEKWAD4BVWFW5D3J5KTDMJEM7I4LRRGGTJ63VSZMTIUX2R4F5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.531477747 (514h7m43.402560792s) [noise=true , address=145.79.227.37:4567 , piece_id=265N6AJVRXBUU34WRXJUG3SOP5D7HVCNG2TLMJZCTX2JGVVG3ULQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541361136 (514h7m43.412444181s) [noise=true , address=145.79.227.5:4222 , piece_id=WSACHEUYFUMHK37G2LKRM2EW7W2CUWQILOB6Y3HHZ5C24TMESDTQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.541452310 (514h7m43.412535355s) [noise=true , address=145.79.227.46:4602 , piece_id=SFVY55YLWT3UV3KRLZJHRGLYX46DPHQT6QOY6WOSL5AVRP3RAOGA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.547434571 (514h7m43.418517606s) [noise=true , address=145.79.227.42:4123 , piece_id=JAABMY2VBHBF7ORF4WLBZWZJWD47OGKR2EHPOT4TNAKCJC32F5KQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.761126180 (514h7m43.632209215s) [noise=true , address=45.91.116.14:4192 , piece_id=BERQV36B2KPFLJXBN4SGO7SETIDW6HDYPXGCP5N2AXPS3VYTGLSQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792487124 (514h7m43.663570159s) [noise=true , address=45.91.116.8:4047 , piece_id=JAYLFTX6DLVDXWFWOVFPETNLJGLRKUQQ4Y7BYKHIXE45UWIQEBXA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.792976496 (514h7m43.664059541s) [noise=true , address=77.90.31.16:4736 , piece_id=BQZ37TCE3VYDJTBXGHMH3KF3IHO5VK3SV5BSF7I44OUV3YRYAQRA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.806870573 (514h7m43.677953608s) [noise=true , address=77.90.31.11:4556 , piece_id=YNJCGBJ3NPFLHHTEYVJ5DSLUS6YWEET2Q2LBSX3GQKFLFBAHIWAA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.824405031 (514h7m43.695488066s) [noise=true , address=45.91.116.7:4011 , piece_id=CSNB7PUZD5YU4MWMM4ZF2VYMNLA65JXZPO7CM7ADV6I5X2NXARMA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.856319741 (514h7m43.727402786s) [noise=true , address=45.91.116.5:4299 , piece_id=IVIKWYBPGR27ZWAPSIZA4Q57QFG3KDMCS4RSA3DXLEV24I324DMQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879719408 (514h7m43.750802453s) [noise=true , address=45.91.116.4:4939 , piece_id=QOYXBHAH4HGURO3YKTXEQQM7T7SA64LGINF2DYYOQ654YRTFAT5Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: <nil> Dec 27 01:36:43.879739525 (514h7m43.750822561s) [seg_pos={0 0}, upload=451146]: Segment reached optimal threshold of 13 pieces. Dec 27 01:36:43.879906464 (514h7m43.750989499s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.17:4971: use of closed network connection Dec 27 01:36:43.879912375 (514h7m43.75099542s) [noise=true , address=145.79.227.17:4971 , piece_id=PCZXO3GVQJ3REY4OBBBL2YWV6J7WPGR5PATBIRY2IMIDHMT4SW2Q, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880028446 (514h7m43.751111491s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 01:36:43.880046109 (514h7m43.751129145s) [noise=true , address=145.79.227.44:4193 , piece_id=6JLEZQ74GX3S7DANOBVRFXXMZRTQGRMERKT6MQSHZJC6GLMPXBZQ, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880063112 (514h7m43.751146147s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54346->45.91.116.9:4262: use of closed network connection Dec 27 01:36:43.880072330 (514h7m43.751155365s) [noise=true , address=45.91.116.9:4262 , piece_id=AP2FOPKDVPXCZYSKGKFIC6R2QYTPK43S2HDPJ5R7UB2BRMVEFANA, seg_pos={0 0}, upload=451146]: Long tail cancellation due to optimal threshold Dec 27 01:36:43.880121062 (514h7m43.751204097s) [seg_pos={0 0}, upload=451146]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:36:43.880230500 (514h7m43.751313535s) [upload=451146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:36:43.880239648 (514h7m43.751322724s) [upload=451146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:36:44.172297988 (514h7m44.043381023s) [upload=451146]: Done uploading segments... Dec 27 01:36:44.172314539 (514h7m44.043397584s) [upload=451146]: Upload finished. err: <nil> Duration: 30.004270878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..XX] duration: 30.003367226s trace: https://jaeger.dev.storj.io/trace/1c660901775cb2a4 FAIL (2025-12-27 01:36:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2S5WW2LU5VZPKL4O5H2LR7I4QY4SSBPG7KO3NKLCXZDYT4J2LNGA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: F44KC5UFDJJYGUDMCPJUZPLMVNFGOFZ6KHPLKGWKLQEHUVXMYHDA) context deadline exceeded ======LOGS====== Dec 27 01:35:45.975763098 (514h6m45.846846143s) [upload=451133]: Starting upload Dec 27 01:35:45.975815569 (514h6m45.846898604s) [upload=451133]: Done starting upload Dec 27 01:35:45.975928894 (514h6m45.847011939s) [upload=451133]: Uploading segments... Dec 27 01:35:45.976015760 (514h6m45.847098805s) [upload=451133]: Got next segment. Inline: false Dec 27 01:35:45.976075042 (514h6m45.847158077s) [upload=451133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:35:45.976082076 (514h6m45.847165111s) [upload=451133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:35:46.221161912 (514h6m46.092244967s) [seg_pos={0 0}, upload=451133]: Begin upload segment... Dec 27 01:35:46.221228969 (514h6m46.092312004s) [seg_pos={0 0}, upload=451133]: Done begin upload segment. Dec 27 01:35:46.221253617 (514h6m46.092336692s) [noise=true , address=145.79.227.16:4911 , piece_id=XETV5XKFQ7WOJQW6S4XBXOF3HGUB6HFZZZY3BYIQJEZTGWVQUADQ, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.221261752 (514h6m46.092344787s) [noise=true , address=145.79.227.35:4481 , piece_id=RMX24HDXVGGQZ6VW7V54RET33GI3D6BOHJGZ3NMSGKGMV42GDJJA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.221335412 (514h6m46.092418457s) [noise=true , address=45.91.116.10:4083 , piece_id=B2V5G56R4XEUWY6V6R5CSQPE54FLGJAZO5NC47ERWF5T5BSYYCFQ, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.221987223 (514h6m46.093070319s) [noise=true , address=77.90.31.17:4773 , piece_id=F44KC5UFDJJYGUDMCPJUZPLMVNFGOFZ6KHPLKGWKLQEHUVXMYHDA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.222582938 (514h6m46.093665973s) [noise=true , address=145.79.227.15:4874 , piece_id=RKXHWVTVJNKOU56LAQD7STJNBUW2GDAMZHKZDY7XKQGG5SKFU6TQ, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.222966899 (514h6m46.094049944s) [noise=true , address=145.79.227.13:4804 , piece_id=WAWGGJEOQ25SETCRCYQU5LUIXNWQTAWVQN3UYWAGEZREVS4VOYCA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.223335721 (514h6m46.094418757s) [noise=true , address=145.79.227.51:4409 , piece_id=SMMW24C7VFDAH4JI6NX6QCCE4SYZXGL3NNLPUH556QBLKEQJWFDA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.223721907 (514h6m46.094804952s) [noise=true , address=145.79.227.50:4374 , piece_id=6U2FJ7NI4XWYXQHHJNFZU3HDQRUJRUXOFHLVUSNEOBI5Q6O2OMGA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.224152777 (514h6m46.095235812s) [noise=true , address=145.79.227.47:4267 , piece_id=LH22776FFIVN5EXSTXDGVHQYINBULXOCWRXZ5ORDU54S6JLY7Z7A, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.224620167 (514h6m46.095703212s) [noise=true , address=145.79.227.14:4838 , piece_id=HU7PGPBRSW6PCCY5HS3QZDR6BJWLQ3J4IDY6QCJJ5AZC6HZUXPTA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.225104128 (514h6m46.096187163s) [noise=true , address=77.90.31.13:4627 , piece_id=27BAP5RBUMDAVBGTAXGZEASBXRG4CYX2DV2LAVEDJNFWUGQH4EDA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.226517760 (514h6m46.097600805s) [noise=true , address=145.79.227.37:4565 , piece_id=UR66GUVSGBFOOAUF2YUTABW4Q6PK4L4QKNBBHEOBOJDYPM75D3CA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.226585048 (514h6m46.097668083s) [noise=true , address=145.79.227.49:4338 , piece_id=IF5LZBSRVI47WXIGSWCYJ7R5NWD4H7TWHOBI4GXRB5QUZCU4ZUNQ, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.227988881 (514h6m46.099071916s) [noise=true , address=45.91.116.13:4154 , piece_id=2S5WW2LU5VZPKL4O5H2LR7I4QY4SSBPG7KO3NKLCXZDYT4J2LNGA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.230348404 (514h6m46.101431459s) [noise=true , address=45.91.116.4:4937 , piece_id=CRY4Y5ODXB7KKE5M2RHYIFLVAREJNUHK2DUR2VLHRGY26TRDVLJQ, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:46.230883914 (514h6m46.101966949s) [noise=true , address=45.91.116.9:4262 , piece_id=2J3KPHZFJWWZLUDSX6MBJI266XV4CKDNPXDNHIU3VD7B4ZLLWBHA, seg_pos={0 0}, upload=451133]: Uploading piece... Dec 27 01:35:47.945303447 (514h6m47.816386482s) [upload=451133]: Next returned nil segment Dec 27 01:35:47.945314368 (514h6m47.816397403s) [upload=451133]: Waiting for error group managing segments... Dec 27 01:35:48.232850516 (514h6m48.103933551s) [noise=true , address=145.79.227.35:4481 , piece_id=RMX24HDXVGGQZ6VW7V54RET33GI3D6BOHJGZ3NMSGKGMV42GDJJA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.240924440 (514h6m48.112007475s) [noise=true , address=145.79.227.47:4267 , piece_id=LH22776FFIVN5EXSTXDGVHQYINBULXOCWRXZ5ORDU54S6JLY7Z7A, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.243856633 (514h6m48.114939678s) [noise=true , address=145.79.227.15:4874 , piece_id=RKXHWVTVJNKOU56LAQD7STJNBUW2GDAMZHKZDY7XKQGG5SKFU6TQ, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.258271181 (514h6m48.129354216s) [noise=true , address=145.79.227.13:4804 , piece_id=WAWGGJEOQ25SETCRCYQU5LUIXNWQTAWVQN3UYWAGEZREVS4VOYCA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.258759952 (514h6m48.129842987s) [noise=true , address=145.79.227.49:4338 , piece_id=IF5LZBSRVI47WXIGSWCYJ7R5NWD4H7TWHOBI4GXRB5QUZCU4ZUNQ, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.262096585 (514h6m48.13317962s) [noise=true , address=145.79.227.50:4374 , piece_id=6U2FJ7NI4XWYXQHHJNFZU3HDQRUJRUXOFHLVUSNEOBI5Q6O2OMGA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.263136404 (514h6m48.134219439s) [noise=true , address=145.79.227.51:4409 , piece_id=SMMW24C7VFDAH4JI6NX6QCCE4SYZXGL3NNLPUH556QBLKEQJWFDA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.263611369 (514h6m48.134694414s) [noise=true , address=145.79.227.37:4565 , piece_id=UR66GUVSGBFOOAUF2YUTABW4Q6PK4L4QKNBBHEOBOJDYPM75D3CA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.267662172 (514h6m48.138745207s) [noise=true , address=145.79.227.16:4911 , piece_id=XETV5XKFQ7WOJQW6S4XBXOF3HGUB6HFZZZY3BYIQJEZTGWVQUADQ, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.276041759 (514h6m48.147124794s) [noise=true , address=145.79.227.14:4838 , piece_id=HU7PGPBRSW6PCCY5HS3QZDR6BJWLQ3J4IDY6QCJJ5AZC6HZUXPTA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.419915994 (514h6m48.290999029s) [noise=true , address=77.90.31.17:4773 , piece_id=F44KC5UFDJJYGUDMCPJUZPLMVNFGOFZ6KHPLKGWKLQEHUVXMYHDA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.591683792 (514h6m48.462766847s) [noise=true , address=45.91.116.13:4154 , piece_id=2S5WW2LU5VZPKL4O5H2LR7I4QY4SSBPG7KO3NKLCXZDYT4J2LNGA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.599970421 (514h6m48.471053456s) [noise=true , address=45.91.116.4:4937 , piece_id=CRY4Y5ODXB7KKE5M2RHYIFLVAREJNUHK2DUR2VLHRGY26TRDVLJQ, seg_pos={0 0}, upload=451133]: Done uploading piece. err: <nil> Dec 27 01:35:48.599990659 (514h6m48.471073704s) [seg_pos={0 0}, upload=451133]: Segment reached optimal threshold of 13 pieces. Dec 27 01:35:48.600167046 (514h6m48.471250091s) [noise=true , address=45.91.116.10:4083 , piece_id=B2V5G56R4XEUWY6V6R5CSQPE54FLGJAZO5NC47ERWF5T5BSYYCFQ, seg_pos={0 0}, upload=451133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 01:35:48.600173007 (514h6m48.471256052s) [noise=true , address=45.91.116.10:4083 , piece_id=B2V5G56R4XEUWY6V6R5CSQPE54FLGJAZO5NC47ERWF5T5BSYYCFQ, seg_pos={0 0}, upload=451133]: Long tail cancellation due to optimal threshold Dec 27 01:35:48.600302573 (514h6m48.471385608s) [noise=true , address=45.91.116.9:4262 , piece_id=2J3KPHZFJWWZLUDSX6MBJI266XV4CKDNPXDNHIU3VD7B4ZLLWBHA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48518->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48518->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48518->45.91.116.9:4262: use of closed network connection Dec 27 01:35:48.600310189 (514h6m48.471393224s) [noise=true , address=45.91.116.9:4262 , piece_id=2J3KPHZFJWWZLUDSX6MBJI266XV4CKDNPXDNHIU3VD7B4ZLLWBHA, seg_pos={0 0}, upload=451133]: Long tail cancellation due to optimal threshold Dec 27 01:35:48.600376274 (514h6m48.47145931s) [noise=true , address=77.90.31.13:4627 , piece_id=27BAP5RBUMDAVBGTAXGZEASBXRG4CYX2DV2LAVEDJNFWUGQH4EDA, seg_pos={0 0}, upload=451133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 01:35:48.600383388 (514h6m48.471466423s) [noise=true , address=77.90.31.13:4627 , piece_id=27BAP5RBUMDAVBGTAXGZEASBXRG4CYX2DV2LAVEDJNFWUGQH4EDA, seg_pos={0 0}, upload=451133]: Long tail cancellation due to optimal threshold Dec 27 01:35:48.600398376 (514h6m48.471481421s) [seg_pos={0 0}, upload=451133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:35:48.600463260 (514h6m48.471546295s) [upload=451133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:35:48.600467638 (514h6m48.471550673s) [upload=451133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:35:48.888120359 (514h6m48.759203434s) [upload=451133]: Done uploading segments... Dec 27 01:35:48.888143863 (514h6m48.759226908s) [upload=451133]: Upload finished. err: <nil> Duration: 30.003394447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..X] duration: 30.003963825s trace: https://jaeger.dev.storj.io/trace/7dd69e4fe7380585 FAIL (2025-12-27 01:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:34:52.747202214 (514h5m52.618285259s) [upload=451120]: Starting upload Dec 27 01:34:52.747283178 (514h5m52.618366223s) [upload=451120]: Done starting upload Dec 27 01:34:52.747363241 (514h5m52.618446286s) [upload=451120]: Uploading segments... Dec 27 01:34:52.747491345 (514h5m52.61857438s) [upload=451120]: Got next segment. Inline: false Dec 27 01:34:52.747520420 (514h5m52.618603455s) [upload=451120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:34:52.747524758 (514h5m52.618607803s) [upload=451120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:34:52.997281149 (514h5m52.868364204s) [seg_pos={0 0}, upload=451120]: Begin upload segment... Dec 27 01:34:52.997344349 (514h5m52.868427395s) [seg_pos={0 0}, upload=451120]: Done begin upload segment. Dec 27 01:34:52.997398794 (514h5m52.868481839s) [noise=true , address=145.79.227.15:4874 , piece_id=KXRRLT32TRRWLZNOV5MNLJQ5DQML4G2VWW5VMKQZFVD4DUDA6QJQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:52.997426976 (514h5m52.868510011s) [noise=true , address=45.91.116.13:4153 , piece_id=OGDH43GOJGAQRU4NCGY2U4YMLUM2MNRM6CMNGOYHIJQE7M7I3I4A, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:52.998154011 (514h5m52.869237046s) [noise=true , address=77.90.31.10:4518 , piece_id=FW6IBZBBY7SH2SL425XZ7BEVASQKZXYJ7BLTUDMCXJOEUIU2CTUQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:52.998331579 (514h5m52.869414614s) [noise=true , address=77.90.31.18:4808 , piece_id=LGGQ5H73NU37CF2YXHTRWJBD7JVXTINSX2TTC552UFEKWEN5QEPQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:52.998968382 (514h5m52.870051417s) [noise=true , address=145.79.227.37:4566 , piece_id=I5B3ESPTIIOPQ4UMFKTYROOZPOEBQEOFECS7HKM4IN45NSU42D7Q, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:52.999761261 (514h5m52.870844296s) [noise=true , address=45.91.116.6:4973 , piece_id=4Z7TUNWV73P7BWZMLR7OR3IL6FC4A74YBI445ELN3SMYH2HXFSZA, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.000175670 (514h5m52.871258716s) [noise=true , address=145.79.227.14:4837 , piece_id=HIB7KWGSRBCDMPMQPG4NIK27KCVK7LGM5WF3ARGJW6XFZ3ZEHDHQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.000999520 (514h5m52.872082565s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEOFHWUARCU5K2QJEXUZEQXFHACTGTBZBNJ3YGNUQUUDX72KN6A, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.001519299 (514h5m52.872602344s) [noise=true , address=77.90.31.19:4841 , piece_id=2ODOLJWYZYLHQ3KVDSUE3KWQTQ4GAGDYDRH5LZKQZQX5OX4YSQJA, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.002088072 (514h5m52.873171117s) [noise=true , address=145.79.227.50:4375 , piece_id=OON4M2OUGX6A5W4LXSMAJWUP2PQUVFZBB3QHEBVJNM5JV6B7LEXQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.002242717 (514h5m52.873325762s) [noise=true , address=145.79.227.16:4909 , piece_id=IHUNOOHW4CFJMSITHFJIO4W5DDFSRUAC76HSONN25FKGAGSEV7YQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.003389149 (514h5m52.874472194s) [noise=true , address=145.79.227.23:4545 , piece_id=KYFC5JTN7YB4XJKVPRM2NN2FDI4LYYTMIROJX4NNKIZTE3BFLTCA, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.004712449 (514h5m52.875795494s) [noise=true , address=45.91.116.7:4012 , piece_id=EACVCI4RCDUOILCUJY7756TQB5DJ3TQ23TZQKTLC5OMGIDFLXCBQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.005335946 (514h5m52.876418991s) [noise=true , address=45.91.116.9:4264 , piece_id=JYOBCOKDYUHBY42WDFWVKILXRZLNLSSWI4WI3C237ATUYWCHY76A, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.005952501 (514h5m52.877035546s) [noise=true , address=45.91.116.14:4189 , piece_id=DEQVEY2XKNOSZEXYHBWBSLVD5AB3VOQQ7R43BZD7PWIT67FZJVYQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:53.006250407 (514h5m52.877333452s) [noise=true , address=45.91.116.5:4299 , piece_id=J6PIZTRK7USBHUXYS5BITHP5OJ4Y33NQWUM4HWZOBP4XWZTEMPMQ, seg_pos={0 0}, upload=451120]: Uploading piece... Dec 27 01:34:54.637512145 (514h5m54.508595191s) [upload=451120]: Next returned nil segment Dec 27 01:34:54.637524479 (514h5m54.508607524s) [upload=451120]: Waiting for error group managing segments... Dec 27 01:34:55.011832663 (514h5m54.882915698s) [noise=true , address=145.79.227.50:4375 , piece_id=OON4M2OUGX6A5W4LXSMAJWUP2PQUVFZBB3QHEBVJNM5JV6B7LEXQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.019300363 (514h5m54.890383408s) [noise=true , address=145.79.227.37:4566 , piece_id=I5B3ESPTIIOPQ4UMFKTYROOZPOEBQEOFECS7HKM4IN45NSU42D7Q, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.022043216 (514h5m54.893126251s) [noise=true , address=145.79.227.15:4874 , piece_id=KXRRLT32TRRWLZNOV5MNLJQ5DQML4G2VWW5VMKQZFVD4DUDA6QJQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.023989412 (514h5m54.895072447s) [noise=true , address=145.79.227.16:4909 , piece_id=IHUNOOHW4CFJMSITHFJIO4W5DDFSRUAC76HSONN25FKGAGSEV7YQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.028446829 (514h5m54.899529874s) [noise=true , address=145.79.227.23:4545 , piece_id=KYFC5JTN7YB4XJKVPRM2NN2FDI4LYYTMIROJX4NNKIZTE3BFLTCA, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.168018223 (514h5m55.039101268s) [noise=true , address=77.90.31.10:4518 , piece_id=FW6IBZBBY7SH2SL425XZ7BEVASQKZXYJ7BLTUDMCXJOEUIU2CTUQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.187613304 (514h5m55.05869634s) [noise=true , address=77.90.31.19:4841 , piece_id=2ODOLJWYZYLHQ3KVDSUE3KWQTQ4GAGDYDRH5LZKQZQX5OX4YSQJA, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.237082731 (514h5m55.108165776s) [noise=true , address=45.91.116.14:4189 , piece_id=DEQVEY2XKNOSZEXYHBWBSLVD5AB3VOQQ7R43BZD7PWIT67FZJVYQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.240621530 (514h5m55.111704575s) [noise=true , address=45.91.116.7:4012 , piece_id=EACVCI4RCDUOILCUJY7756TQB5DJ3TQ23TZQKTLC5OMGIDFLXCBQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.250703547 (514h5m55.121786582s) [noise=true , address=45.91.116.9:4264 , piece_id=JYOBCOKDYUHBY42WDFWVKILXRZLNLSSWI4WI3C237ATUYWCHY76A, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.260112593 (514h5m55.131195628s) [noise=true , address=45.91.116.6:4973 , piece_id=4Z7TUNWV73P7BWZMLR7OR3IL6FC4A74YBI445ELN3SMYH2HXFSZA, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.314919423 (514h5m55.186002458s) [noise=true , address=45.91.116.13:4153 , piece_id=OGDH43GOJGAQRU4NCGY2U4YMLUM2MNRM6CMNGOYHIJQE7M7I3I4A, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.957994948 (514h5m55.829077993s) [noise=true , address=145.79.227.14:4837 , piece_id=HIB7KWGSRBCDMPMQPG4NIK27KCVK7LGM5WF3ARGJW6XFZ3ZEHDHQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: <nil> Dec 27 01:34:55.958017441 (514h5m55.829100476s) [seg_pos={0 0}, upload=451120]: Segment reached optimal threshold of 13 pieces. Dec 27 01:34:55.958237059 (514h5m55.829320104s) [noise=true , address=77.90.31.18:4808 , piece_id=LGGQ5H73NU37CF2YXHTRWJBD7JVXTINSX2TTC552UFEKWEN5QEPQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 01:34:55.958243481 (514h5m55.829326526s) [noise=true , address=77.90.31.18:4808 , piece_id=LGGQ5H73NU37CF2YXHTRWJBD7JVXTINSX2TTC552UFEKWEN5QEPQ, seg_pos={0 0}, upload=451120]: Long tail cancellation due to optimal threshold Dec 27 01:34:55.958588658 (514h5m55.829671693s) [noise=true , address=45.91.116.5:4299 , piece_id=J6PIZTRK7USBHUXYS5BITHP5OJ4Y33NQWUM4HWZOBP4XWZTEMPMQ, seg_pos={0 0}, upload=451120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 01:34:55.958596062 (514h5m55.829679107s) [noise=true , address=45.91.116.5:4299 , piece_id=J6PIZTRK7USBHUXYS5BITHP5OJ4Y33NQWUM4HWZOBP4XWZTEMPMQ, seg_pos={0 0}, upload=451120]: Long tail cancellation due to optimal threshold Dec 27 01:34:55.958599819 (514h5m55.829682864s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEOFHWUARCU5K2QJEXUZEQXFHACTGTBZBNJ3YGNUQUUDX72KN6A, seg_pos={0 0}, upload=451120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 01:34:55.958621421 (514h5m55.829704456s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEOFHWUARCU5K2QJEXUZEQXFHACTGTBZBNJ3YGNUQUUDX72KN6A, seg_pos={0 0}, upload=451120]: Long tail cancellation due to optimal threshold Dec 27 01:34:55.958657820 (514h5m55.829740855s) [seg_pos={0 0}, upload=451120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:34:55.958804329 (514h5m55.829887364s) [upload=451120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:34:55.958811843 (514h5m55.829894878s) [upload=451120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:34:56.247505506 (514h5m56.118588551s) [upload=451120]: Done uploading segments... Dec 27 01:34:56.247557515 (514h5m56.11864055s) [upload=451120]: Upload finished. err: <nil> Duration: 30.003990355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX..] duration: 5.766492398s trace: https://jaeger.dev.storj.io/trace/123350ba17c87eef FAIL (2025-12-27 01:33:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ) context deadline exceeded ======LOGS====== 6.370211011s) [upload=451076]: Uploading segments... Dec 27 01:31:46.499429431 (514h2m46.370512476s) [upload=451076]: Got next segment. Inline: false Dec 27 01:31:46.499517688 (514h2m46.370600733s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:31:46.499524842 (514h2m46.370607877s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:31:46.749923983 (514h2m46.621007028s) [seg_pos={0 0}, upload=451076]: Begin upload segment... Dec 27 01:31:46.749982825 (514h2m46.62106587s) [seg_pos={0 0}, upload=451076]: Done begin upload segment. Dec 27 01:31:46.750014696 (514h2m46.621097741s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.754322519 (514h2m46.625405554s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.755542732 (514h2m46.626625777s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.756979808 (514h2m46.628062854s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.757385912 (514h2m46.628468957s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.758859708 (514h2m46.629942753s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.759132478 (514h2m46.630215513s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760644877 (514h2m46.631727922s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760877540 (514h2m46.631960586s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762876066 (514h2m46.633959111s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762912765 (514h2m46.63399581s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.764831399 (514h2m46.635914444s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.765047239 (514h2m46.636130275s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766763949 (514h2m46.637846984s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766797323 (514h2m46.637880368s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.768846053 (514h2m46.639929098s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:48.488884231 (514h2m48.359967276s) [upload=451076]: Next returned nil segment Dec 27 01:31:48.488898148 (514h2m48.359981183s) [upload=451076]: Waiting for error group managing segments... Dec 27 01:31:48.782217374 (514h2m48.653300409s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.789106051 (514h2m48.660189096s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.793690781 (514h2m48.664773816s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.808026999 (514h2m48.679110034s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.815213482 (514h2m48.686296528s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.823864395 (514h2m48.69494744s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.830511131 (514h2m48.701594166s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.012730189 (514h2m48.883813224s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.061897139 (514h2m48.932980184s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.103645794 (514h2m48.974728829s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.108827992 (514h2m48.979911027s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.113947189 (514h2m48.985030234s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410264104 (514h2m49.281347149s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410303388 (514h2m49.281386433s) [seg_pos={0 0}, upload=451076]: Segment reached optimal threshold of 13 pieces. Dec 27 01:31:49.410609150 (514h2m49.281692205s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection Dec 27 01:31:49.410620983 (514h2m49.281704028s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410635961 (514h2m49.281719006s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection Dec 27 01:31:49.410643545 (514h2m49.28172659s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410905675 (514h2m49.28198873s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 01:31:49.410918048 (514h2m49.282001083s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410939560 (514h2m49.282022605s) [seg_pos={0 0}, upload=451076]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:31:49.411071842 (514h2m49.282154887s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:31:49.411079195 (514h2m49.28216224s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:31:49.696655558 (514h2m49.567738594s) [upload=451076]: Done uploading segments... Dec 27 01:31:49.696716274 (514h2m49.567799309s) [upload=451076]: Upload finished. err: <nil> Duration: 5.766513797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX.] duration: 23.693730243s trace: https://jaeger.dev.storj.io/trace/123350ba17c87eef FAIL (2025-12-27 01:33:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ) context deadline exceeded ======LOGS====== 6.370211011s) [upload=451076]: Uploading segments... Dec 27 01:31:46.499429431 (514h2m46.370512476s) [upload=451076]: Got next segment. Inline: false Dec 27 01:31:46.499517688 (514h2m46.370600733s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:31:46.499524842 (514h2m46.370607877s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:31:46.749923983 (514h2m46.621007028s) [seg_pos={0 0}, upload=451076]: Begin upload segment... Dec 27 01:31:46.749982825 (514h2m46.62106587s) [seg_pos={0 0}, upload=451076]: Done begin upload segment. Dec 27 01:31:46.750014696 (514h2m46.621097741s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.754322519 (514h2m46.625405554s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.755542732 (514h2m46.626625777s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.756979808 (514h2m46.628062854s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.757385912 (514h2m46.628468957s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.758859708 (514h2m46.629942753s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.759132478 (514h2m46.630215513s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760644877 (514h2m46.631727922s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760877540 (514h2m46.631960586s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762876066 (514h2m46.633959111s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762912765 (514h2m46.63399581s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.764831399 (514h2m46.635914444s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.765047239 (514h2m46.636130275s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766763949 (514h2m46.637846984s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766797323 (514h2m46.637880368s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.768846053 (514h2m46.639929098s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:48.488884231 (514h2m48.359967276s) [upload=451076]: Next returned nil segment Dec 27 01:31:48.488898148 (514h2m48.359981183s) [upload=451076]: Waiting for error group managing segments... Dec 27 01:31:48.782217374 (514h2m48.653300409s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.789106051 (514h2m48.660189096s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.793690781 (514h2m48.664773816s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.808026999 (514h2m48.679110034s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.815213482 (514h2m48.686296528s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.823864395 (514h2m48.69494744s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.830511131 (514h2m48.701594166s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.012730189 (514h2m48.883813224s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.061897139 (514h2m48.932980184s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.103645794 (514h2m48.974728829s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.108827992 (514h2m48.979911027s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.113947189 (514h2m48.985030234s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410264104 (514h2m49.281347149s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410303388 (514h2m49.281386433s) [seg_pos={0 0}, upload=451076]: Segment reached optimal threshold of 13 pieces. Dec 27 01:31:49.410609150 (514h2m49.281692205s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection Dec 27 01:31:49.410620983 (514h2m49.281704028s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410635961 (514h2m49.281719006s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection Dec 27 01:31:49.410643545 (514h2m49.28172659s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410905675 (514h2m49.28198873s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 01:31:49.410918048 (514h2m49.282001083s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410939560 (514h2m49.282022605s) [seg_pos={0 0}, upload=451076]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:31:49.411071842 (514h2m49.282154887s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:31:49.411079195 (514h2m49.28216224s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:31:49.696655558 (514h2m49.567738594s) [upload=451076]: Done uploading segments... Dec 27 01:31:49.696716274 (514h2m49.567799309s) [upload=451076]: Upload finished. err: <nil> Duration: 23.693770789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [.X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.XX] duration: 30.002787747s trace: https://jaeger.dev.storj.io/trace/123350ba17c87eef FAIL (2025-12-27 01:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ) context deadline exceeded ======LOGS====== 6.370211011s) [upload=451076]: Uploading segments... Dec 27 01:31:46.499429431 (514h2m46.370512476s) [upload=451076]: Got next segment. Inline: false Dec 27 01:31:46.499517688 (514h2m46.370600733s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:31:46.499524842 (514h2m46.370607877s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:31:46.749923983 (514h2m46.621007028s) [seg_pos={0 0}, upload=451076]: Begin upload segment... Dec 27 01:31:46.749982825 (514h2m46.62106587s) [seg_pos={0 0}, upload=451076]: Done begin upload segment. Dec 27 01:31:46.750014696 (514h2m46.621097741s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.754322519 (514h2m46.625405554s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.755542732 (514h2m46.626625777s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.756979808 (514h2m46.628062854s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.757385912 (514h2m46.628468957s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.758859708 (514h2m46.629942753s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.759132478 (514h2m46.630215513s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760644877 (514h2m46.631727922s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.760877540 (514h2m46.631960586s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762876066 (514h2m46.633959111s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.762912765 (514h2m46.63399581s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.764831399 (514h2m46.635914444s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.765047239 (514h2m46.636130275s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766763949 (514h2m46.637846984s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.766797323 (514h2m46.637880368s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:46.768846053 (514h2m46.639929098s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Uploading piece... Dec 27 01:31:48.488884231 (514h2m48.359967276s) [upload=451076]: Next returned nil segment Dec 27 01:31:48.488898148 (514h2m48.359981183s) [upload=451076]: Waiting for error group managing segments... Dec 27 01:31:48.782217374 (514h2m48.653300409s) [noise=true , address=145.79.227.45:4230 , piece_id=X766B4XDPDZGPAWLZQO5MJW2VFHDQCUKI75BYDVYNCISPSIGAFLQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.789106051 (514h2m48.660189096s) [noise=true , address=145.79.227.15:4875 , piece_id=PXJLUPIAHIAIOLYPQDXEXJU2QF7DOHZCR3XG3FDN3VVVDQBOBYCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.793690781 (514h2m48.664773816s) [noise=true , address=145.79.227.29:4941 , piece_id=MBLXU2SSGSKS44ZCETGMEQTROHY4EPP42Y6XYA43P7B52COJEGMA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.808026999 (514h2m48.679110034s) [noise=true , address=145.79.227.52:4446 , piece_id=R7TQETQB2CCLSWFCEIUDSSPVWTWN3KQSG66KW7NYARKESX6HQXCQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.815213482 (514h2m48.686296528s) [noise=true , address=145.79.227.42:4122 , piece_id=3KB75ZOTBNGNP2BVTE5O5OGPFE5ASNYGF5RU4WXXTUK2Z4OKCGQA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.823864395 (514h2m48.69494744s) [noise=true , address=145.79.227.48:4301 , piece_id=JIFAP33RJ3LP2DZYFFLTX7GNGAIINT4M4KEJC4NBGKZOWSNSCSIA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:48.830511131 (514h2m48.701594166s) [noise=true , address=145.79.227.47:4267 , piece_id=MBZMX2BSSQ6YUO5NPGIVYMWD25LFJS2L6EF5LTGN3OJHTN2WEE4A, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.012730189 (514h2m48.883813224s) [noise=true , address=77.90.31.10:4518 , piece_id=PXMNVIYAASQJMCDR7JQULIY7ETKSB2UAZEKJES57LNYQICMGYJXQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.061897139 (514h2m48.932980184s) [noise=true , address=45.91.116.4:4940 , piece_id=E7E4CAUYQE6TEI2O3WSLTAWQ2ZSGW3ZMRUPYXISINFYHNWEFOODA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.103645794 (514h2m48.974728829s) [noise=true , address=45.91.116.9:4264 , piece_id=MQP6SBFQAJ5XGPVHV3P345DVLG57GQ5PYMCZLYRDXHNUVNYXUYVA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.108827992 (514h2m48.979911027s) [noise=true , address=45.91.116.5:4298 , piece_id=XSVOLNR7BIL5LKKZEGP6NR4K4UA5SVSRMRQOTO566GK2Y2YUM2QQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.113947189 (514h2m48.985030234s) [noise=true , address=45.91.116.10:4082 , piece_id=SO6UC5R4UVAM63IA6PZCNA5HXXNJTDCB73A23UBU3FPQHX3TD2UA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410264104 (514h2m49.281347149s) [noise=true , address=45.91.116.13:4153 , piece_id=GS6NOA3A4K5VZOLDR3WDIB7VL3PHOI5QGNALNLXQQUOADAJPZ76Q, seg_pos={0 0}, upload=451076]: Done uploading piece. err: <nil> Dec 27 01:31:49.410303388 (514h2m49.281386433s) [seg_pos={0 0}, upload=451076]: Segment reached optimal threshold of 13 pieces. Dec 27 01:31:49.410609150 (514h2m49.281692205s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->145.79.227.17:4971: use of closed network connection Dec 27 01:31:49.410620983 (514h2m49.281704028s) [noise=true , address=145.79.227.17:4971 , piece_id=6T2MW5TB36E7HQ6PQKAHOOGRVNQMFTISBS3NVKCE7POVTXBZGUHA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410635961 (514h2m49.281719006s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54768->45.91.116.14:4191: use of closed network connection Dec 27 01:31:49.410643545 (514h2m49.28172659s) [noise=true , address=45.91.116.14:4191 , piece_id=OJFSLYZVW2D5YH6FQOXGQTVOJCQI2BIGGPRFSK743K3AJTCIJPIQ, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410905675 (514h2m49.28198873s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 01:31:49.410918048 (514h2m49.282001083s) [noise=true , address=145.79.227.44:4195 , piece_id=KCIUSUVPCFQWOUNTEELIKEGR3XRGTR23UJAN3CJEUU3RWWREKFRA, seg_pos={0 0}, upload=451076]: Long tail cancellation due to optimal threshold Dec 27 01:31:49.410939560 (514h2m49.282022605s) [seg_pos={0 0}, upload=451076]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:31:49.411071842 (514h2m49.282154887s) [upload=451076]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:31:49.411079195 (514h2m49.28216224s) [upload=451076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:31:49.696655558 (514h2m49.567738594s) [upload=451076]: Done uploading segments... Dec 27 01:31:49.696716274 (514h2m49.567799309s) [upload=451076]: Upload finished. err: <nil> Duration: 30.002800171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [..X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.X] duration: 30.003129583s trace: https://jaeger.dev.storj.io/trace/eadc0a3979dc63d FAIL (2025-12-27 01:31:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:31:16.254790162 (514h2m16.125873207s) [upload=451068]: Starting upload Dec 27 01:31:16.254844886 (514h2m16.125927931s) [upload=451068]: Done starting upload Dec 27 01:31:16.254990984 (514h2m16.126074029s) [upload=451068]: Uploading segments... Dec 27 01:31:16.255082649 (514h2m16.126165684s) [upload=451068]: Got next segment. Inline: false Dec 27 01:31:16.255124618 (514h2m16.126207663s) [upload=451068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:31:16.255131020 (514h2m16.126214066s) [upload=451068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:31:16.500652766 (514h2m16.371735801s) [seg_pos={0 0}, upload=451068]: Begin upload segment... Dec 27 01:31:16.500713973 (514h2m16.371797018s) [seg_pos={0 0}, upload=451068]: Done begin upload segment. Dec 27 01:31:16.500737187 (514h2m16.371820232s) [noise=true , address=145.79.227.51:4409 , piece_id=YGP75MWBYS6GHUPLDRBLZHGTQ25LL7WYMTGBENAXJEVGLVJ7UBOQ, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.500769859 (514h2m16.371852894s) [noise=true , address=77.90.31.11:4556 , piece_id=MHJXMUEICZJG4QJLXJSLBJBYG2PITRRHNWTUWTOKZFU6PECIYRXA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.501310068 (514h2m16.372393103s) [noise=true , address=145.79.227.36:4518 , piece_id=DR4CGVXQ77EUG56HK4XFVWDHTSVIRMN23HDI5PL56HHGNO7Q2VXQ, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.501679621 (514h2m16.372762666s) [noise=true , address=145.79.227.19:4077 , piece_id=XU6EVQY3ZGJVI72EXULRRMJDSQFP64LFSIKWYYGOSOUASV63RSYQ, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.502131522 (514h2m16.373214557s) [noise=true , address=145.79.227.15:4875 , piece_id=PQQ5GE4MK6X6AHPI73BD4Q47ASKTPL67VOOKL56YXR5RMMLDBLGA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.502550068 (514h2m16.373633114s) [noise=true , address=145.79.227.3:4151 , piece_id=JQLUIORSWCTSHWAHHNLN3A2DMPWH4WKR5OB6Z34EZJ7HIGWMYDDA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.502932597 (514h2m16.374015642s) [noise=true , address=77.90.31.19:4844 , piece_id=YU6G4IWYE7YMPYAZ3G6SHBHYWA5NFHKYBCLYYPUPGWAL5XZJE43Q, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.503272865 (514h2m16.37435591s) [noise=true , address=145.79.227.45:4229 , piece_id=SS27QWUZX2UC34VWESJ3QALCT2JB5DAFB6EIOYIVXQYMQV5QMLUA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.503662998 (514h2m16.374746033s) [noise=true , address=145.79.227.6:4257 , piece_id=BOY5BRVRCQ3GJ353K3ZV7RTJ4BDZGAMXFYLMMGFXSYYY7EV4VKQA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.505082000 (514h2m16.376165045s) [noise=true , address=45.91.116.12:4225 , piece_id=JXWJPURLUDIXBSZG2OLMOPEPOBV4XC45HTHGSRGQMMFADSEZV7CQ, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.505097178 (514h2m16.376180214s) [noise=true , address=45.91.116.7:4011 , piece_id=WUAVFV5HI3IX2NYKINHUSV27X6E3EIXPF6KCPTXXWOSSSE245DAQ, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.506582227 (514h2m16.377665272s) [noise=true , address=145.79.227.52:4447 , piece_id=BHJN272MABAYEPVG2GWPJ6CIJ56ZNFS2XXN4UYP4RDYEWH2QUMMA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.507264957 (514h2m16.378348002s) [noise=true , address=77.90.31.15:4700 , piece_id=XYQZD4JUS62YFAX4EKY73TQDFDWNCXR2F2FWEBY43QXHQC6GGSVA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.509186055 (514h2m16.3802691s) [noise=true , address=145.79.227.14:4838 , piece_id=A63FX33YVRRV45IJZESCX4FDHSJMO4C6PX5HXI55ELTK2FBBEI7A, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.516180583 (514h2m16.387263628s) [noise=true , address=145.79.227.12:4767 , piece_id=Q4XJDWZU6FB6WK36HB2IM6EFYKSEWQU7A7SRIC4OZ4MOE4YBPLRA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:16.520898366 (514h2m16.391981401s) [noise=true , address=145.79.227.48:4302 , piece_id=EYDZ3MCFWZTRV5FHVDJTNSEXHT4LQYZWZOALALK3TWK6FPNF2PFA, seg_pos={0 0}, upload=451068]: Uploading piece... Dec 27 01:31:18.218822396 (514h2m18.089905442s) [upload=451068]: Next returned nil segment Dec 27 01:31:18.218836384 (514h2m18.089919429s) [upload=451068]: Waiting for error group managing segments... Dec 27 01:31:18.493718774 (514h2m18.364801819s) [noise=true , address=145.79.227.15:4875 , piece_id=PQQ5GE4MK6X6AHPI73BD4Q47ASKTPL67VOOKL56YXR5RMMLDBLGA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.514139588 (514h2m18.385222623s) [noise=true , address=145.79.227.52:4447 , piece_id=BHJN272MABAYEPVG2GWPJ6CIJ56ZNFS2XXN4UYP4RDYEWH2QUMMA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.521353584 (514h2m18.392436619s) [noise=true , address=145.79.227.14:4838 , piece_id=A63FX33YVRRV45IJZESCX4FDHSJMO4C6PX5HXI55ELTK2FBBEI7A, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.524037896 (514h2m18.395120941s) [noise=true , address=145.79.227.6:4257 , piece_id=BOY5BRVRCQ3GJ353K3ZV7RTJ4BDZGAMXFYLMMGFXSYYY7EV4VKQA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.529028418 (514h2m18.400111453s) [noise=true , address=145.79.227.45:4229 , piece_id=SS27QWUZX2UC34VWESJ3QALCT2JB5DAFB6EIOYIVXQYMQV5QMLUA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.530958313 (514h2m18.402041348s) [noise=true , address=145.79.227.3:4151 , piece_id=JQLUIORSWCTSHWAHHNLN3A2DMPWH4WKR5OB6Z34EZJ7HIGWMYDDA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.533661330 (514h2m18.404744375s) [noise=true , address=145.79.227.51:4409 , piece_id=YGP75MWBYS6GHUPLDRBLZHGTQ25LL7WYMTGBENAXJEVGLVJ7UBOQ, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.540629056 (514h2m18.411712092s) [noise=true , address=145.79.227.36:4518 , piece_id=DR4CGVXQ77EUG56HK4XFVWDHTSVIRMN23HDI5PL56HHGNO7Q2VXQ, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.541038446 (514h2m18.412121491s) [noise=true , address=145.79.227.48:4302 , piece_id=EYDZ3MCFWZTRV5FHVDJTNSEXHT4LQYZWZOALALK3TWK6FPNF2PFA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.546070368 (514h2m18.417153403s) [noise=true , address=145.79.227.12:4767 , piece_id=Q4XJDWZU6FB6WK36HB2IM6EFYKSEWQU7A7SRIC4OZ4MOE4YBPLRA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.788184193 (514h2m18.659267228s) [noise=true , address=77.90.31.19:4844 , piece_id=YU6G4IWYE7YMPYAZ3G6SHBHYWA5NFHKYBCLYYPUPGWAL5XZJE43Q, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:18.845206208 (514h2m18.716289244s) [noise=true , address=45.91.116.12:4225 , piece_id=JXWJPURLUDIXBSZG2OLMOPEPOBV4XC45HTHGSRGQMMFADSEZV7CQ, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:19.561851619 (514h2m19.432934654s) [noise=true , address=45.91.116.7:4011 , piece_id=WUAVFV5HI3IX2NYKINHUSV27X6E3EIXPF6KCPTXXWOSSSE245DAQ, seg_pos={0 0}, upload=451068]: Done uploading piece. err: <nil> Dec 27 01:31:19.561871778 (514h2m19.432954813s) [seg_pos={0 0}, upload=451068]: Segment reached optimal threshold of 13 pieces. Dec 27 01:31:19.562061128 (514h2m19.433144163s) [noise=true , address=145.79.227.19:4077 , piece_id=XU6EVQY3ZGJVI72EXULRRMJDSQFP64LFSIKWYYGOSOUASV63RSYQ, seg_pos={0 0}, upload=451068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:36354->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:36354->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36354->145.79.227.19:4077: use of closed network connection Dec 27 01:31:19.562067281 (514h2m19.433150316s) [noise=true , address=145.79.227.19:4077 , piece_id=XU6EVQY3ZGJVI72EXULRRMJDSQFP64LFSIKWYYGOSOUASV63RSYQ, seg_pos={0 0}, upload=451068]: Long tail cancellation due to optimal threshold Dec 27 01:31:19.562195464 (514h2m19.433278499s) [noise=true , address=77.90.31.11:4556 , piece_id=MHJXMUEICZJG4QJLXJSLBJBYG2PITRRHNWTUWTOKZFU6PECIYRXA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 01:31:19.562213679 (514h2m19.433296714s) [noise=true , address=77.90.31.11:4556 , piece_id=MHJXMUEICZJG4QJLXJSLBJBYG2PITRRHNWTUWTOKZFU6PECIYRXA, seg_pos={0 0}, upload=451068]: Long tail cancellation due to optimal threshold Dec 27 01:31:19.562235240 (514h2m19.433318275s) [noise=true , address=77.90.31.15:4700 , piece_id=XYQZD4JUS62YFAX4EKY73TQDFDWNCXR2F2FWEBY43QXHQC6GGSVA, seg_pos={0 0}, upload=451068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 01:31:19.562242434 (514h2m19.433325479s) [noise=true , address=77.90.31.15:4700 , piece_id=XYQZD4JUS62YFAX4EKY73TQDFDWNCXR2F2FWEBY43QXHQC6GGSVA, seg_pos={0 0}, upload=451068]: Long tail cancellation due to optimal threshold Dec 27 01:31:19.562259396 (514h2m19.433342441s) [seg_pos={0 0}, upload=451068]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:31:19.562372601 (514h2m19.433455646s) [upload=451068]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:31:19.562440230 (514h2m19.433523265s) [upload=451068]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:31:19.852885576 (514h2m19.723968611s) [upload=451068]: Done uploading segments... Dec 27 01:31:19.852913609 (514h2m19.723996644s) [upload=451068]: Upload finished. err: <nil> Duration: 30.003152528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [...X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX.] duration: 6.056020178s trace: https://jaeger.dev.storj.io/trace/71fa278993a6ae51 FAIL (2025-12-27 01:30:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ) context deadline exceeded ======LOGS====== 1m13.492825862s) [upload=451052]: Uploading segments... Dec 27 01:30:13.621791390 (514h1m13.492874455s) [upload=451052]: Got next segment. Inline: false Dec 27 01:30:13.621873356 (514h1m13.492956391s) [upload=451052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:30:13.621881701 (514h1m13.492964736s) [upload=451052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:30:13.866269556 (514h1m13.737352591s) [seg_pos={0 0}, upload=451052]: Begin upload segment... Dec 27 01:30:13.866312919 (514h1m13.737395954s) [seg_pos={0 0}, upload=451052]: Done begin upload segment. Dec 27 01:30:13.866342916 (514h1m13.737425971s) [noise=true , address=145.79.227.48:4303 , piece_id=AGUTO6PWOVIJPDQ5X234CLSZOIG7QH4WLVTENFGWSG2NKGRJXKWA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.866382621 (514h1m13.737465667s) [noise=true , address=45.91.116.5:4300 , piece_id=SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.866865411 (514h1m13.737948456s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867144001 (514h1m13.738227036s) [noise=true , address=77.90.31.10:4520 , piece_id=IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867500791 (514h1m13.738583826s) [noise=true , address=45.91.116.7:4010 , piece_id=5SKXCDDP347KXNV36CYJDSP4FD52NL2NN6OLWINJRXC3EZWBCS2Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867888991 (514h1m13.738972036s) [noise=true , address=145.79.227.46:4602 , piece_id=JGGNTS27I6BF7YAE2DK2GC3QVGE6VOI3EETAGXHT3KTI63B7TLJA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868221132 (514h1m13.739304168s) [noise=true , address=77.90.31.11:4556 , piece_id=WO47NGFXL44MBKVHJHJ7RYGOOLJFRA25V553R6JI2AWDVKVS4K2Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868508149 (514h1m13.739591184s) [noise=true , address=145.79.227.5:4221 , piece_id=64X2B5A2SQNLSYXIR3RWWZFBS22MURKVKFMVDOIZ4YKAPC3GRPWQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868813891 (514h1m13.739896936s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.870237332 (514h1m13.741320367s) [noise=true , address=145.79.227.14:4838 , piece_id=EELRNX7PWV4RCJNL2VM276BISNYGHTYDNOCSINOJQGYR2VYKZ3XA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.871587683 (514h1m13.742670718s) [noise=true , address=45.91.116.4:4940 , piece_id=ULDHQXBYQVAJGFQGPTMO6MGSEFRULSLP5KKB7TMBMBK55BOZBJCA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.872935639 (514h1m13.744018684s) [noise=true , address=145.79.227.41:4087 , piece_id=TAFBPAMWTHKFYAYYUNUY6N4J75AK7IVOLREEOGRXT2MOTDAUUQKQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.874270802 (514h1m13.745353847s) [noise=true , address=145.79.227.51:4410 , piece_id=ML5EDUBWPA6HUXPENG22JPT3N7L6CP6YMVF7VN4VF26KSDUMXFZQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.874771235 (514h1m13.74585428s) [noise=true , address=145.79.227.45:4229 , piece_id=YNST54V5BWHMJNJXDENSHC4L6USCKCJGJLAOGUJNTXTXKNAPU4LA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.876045380 (514h1m13.747128426s) [noise=true , address=145.79.227.50:4375 , piece_id=QXYI5OVTTGPXJAJA74JQKIS7NGCKFLJHZFZX25SM2KRPJWJKRHBA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.879870645 (514h1m13.75095368s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:15.568895479 (514h1m15.439978524s) [upload=451052]: Next returned nil segment Dec 27 01:30:15.568912212 (514h1m15.439995257s) [upload=451052]: Waiting for error group managing segments... Dec 27 01:30:15.870087627 (514h1m15.741170662s) [noise=true , address=145.79.227.46:4602 , piece_id=JGGNTS27I6BF7YAE2DK2GC3QVGE6VOI3EETAGXHT3KTI63B7TLJA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.882024686 (514h1m15.753107721s) [noise=true , address=145.79.227.45:4229 , piece_id=YNST54V5BWHMJNJXDENSHC4L6USCKCJGJLAOGUJNTXTXKNAPU4LA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.888189796 (514h1m15.759272831s) [noise=true , address=145.79.227.5:4221 , piece_id=64X2B5A2SQNLSYXIR3RWWZFBS22MURKVKFMVDOIZ4YKAPC3GRPWQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.890313279 (514h1m15.761396314s) [noise=true , address=145.79.227.50:4375 , piece_id=QXYI5OVTTGPXJAJA74JQKIS7NGCKFLJHZFZX25SM2KRPJWJKRHBA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.913201270 (514h1m15.784284315s) [noise=true , address=145.79.227.48:4303 , piece_id=AGUTO6PWOVIJPDQ5X234CLSZOIG7QH4WLVTENFGWSG2NKGRJXKWA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.913495611 (514h1m15.784578656s) [noise=true , address=145.79.227.51:4410 , piece_id=ML5EDUBWPA6HUXPENG22JPT3N7L6CP6YMVF7VN4VF26KSDUMXFZQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.915460122 (514h1m15.786543167s) [noise=true , address=145.79.227.14:4838 , piece_id=EELRNX7PWV4RCJNL2VM276BISNYGHTYDNOCSINOJQGYR2VYKZ3XA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.044107473 (514h1m15.915190518s) [noise=true , address=77.90.31.10:4520 , piece_id=IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.065944493 (514h1m15.937027538s) [noise=true , address=77.90.31.11:4556 , piece_id=WO47NGFXL44MBKVHJHJ7RYGOOLJFRA25V553R6JI2AWDVKVS4K2Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.097550454 (514h1m15.968633489s) [noise=true , address=45.91.116.7:4010 , piece_id=5SKXCDDP347KXNV36CYJDSP4FD52NL2NN6OLWINJRXC3EZWBCS2Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.131645786 (514h1m16.002728831s) [noise=true , address=45.91.116.5:4300 , piece_id=SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.231608708 (514h1m16.102691753s) [noise=true , address=45.91.116.4:4940 , piece_id=ULDHQXBYQVAJGFQGPTMO6MGSEFRULSLP5KKB7TMBMBK55BOZBJCA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.304258012 (514h1m16.175341047s) [noise=true , address=145.79.227.41:4087 , piece_id=TAFBPAMWTHKFYAYYUNUY6N4J75AK7IVOLREEOGRXT2MOTDAUUQKQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.304277859 (514h1m16.175360894s) [seg_pos={0 0}, upload=451052]: Segment reached optimal threshold of 13 pieces. Dec 27 01:30:16.304495934 (514h1m16.175578969s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection Dec 27 01:30:16.304502487 (514h1m16.175585533s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304708971 (514h1m16.175792016s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 01:30:16.304716435 (514h1m16.17579947s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304867101 (514h1m16.175950156s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection Dec 27 01:30:16.304889835 (514h1m16.17597288s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304942595 (514h1m16.17602564s) [seg_pos={0 0}, upload=451052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:30:16.305064067 (514h1m16.176147112s) [upload=451052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:30:16.305071951 (514h1m16.176155007s) [upload=451052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:30:16.589704007 (514h1m16.460787052s) [upload=451052]: Done uploading segments... Dec 27 01:30:16.589729295 (514h1m16.46081234s) [upload=451052]: Upload finished. err: <nil> Duration: 6.056044466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XXX] duration: 30.003232958s trace: https://jaeger.dev.storj.io/trace/71fa278993a6ae51 FAIL (2025-12-27 01:30:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ) context deadline exceeded ======LOGS====== 1m13.492825862s) [upload=451052]: Uploading segments... Dec 27 01:30:13.621791390 (514h1m13.492874455s) [upload=451052]: Got next segment. Inline: false Dec 27 01:30:13.621873356 (514h1m13.492956391s) [upload=451052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:30:13.621881701 (514h1m13.492964736s) [upload=451052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:30:13.866269556 (514h1m13.737352591s) [seg_pos={0 0}, upload=451052]: Begin upload segment... Dec 27 01:30:13.866312919 (514h1m13.737395954s) [seg_pos={0 0}, upload=451052]: Done begin upload segment. Dec 27 01:30:13.866342916 (514h1m13.737425971s) [noise=true , address=145.79.227.48:4303 , piece_id=AGUTO6PWOVIJPDQ5X234CLSZOIG7QH4WLVTENFGWSG2NKGRJXKWA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.866382621 (514h1m13.737465667s) [noise=true , address=45.91.116.5:4300 , piece_id=SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.866865411 (514h1m13.737948456s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867144001 (514h1m13.738227036s) [noise=true , address=77.90.31.10:4520 , piece_id=IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867500791 (514h1m13.738583826s) [noise=true , address=45.91.116.7:4010 , piece_id=5SKXCDDP347KXNV36CYJDSP4FD52NL2NN6OLWINJRXC3EZWBCS2Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.867888991 (514h1m13.738972036s) [noise=true , address=145.79.227.46:4602 , piece_id=JGGNTS27I6BF7YAE2DK2GC3QVGE6VOI3EETAGXHT3KTI63B7TLJA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868221132 (514h1m13.739304168s) [noise=true , address=77.90.31.11:4556 , piece_id=WO47NGFXL44MBKVHJHJ7RYGOOLJFRA25V553R6JI2AWDVKVS4K2Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868508149 (514h1m13.739591184s) [noise=true , address=145.79.227.5:4221 , piece_id=64X2B5A2SQNLSYXIR3RWWZFBS22MURKVKFMVDOIZ4YKAPC3GRPWQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.868813891 (514h1m13.739896936s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.870237332 (514h1m13.741320367s) [noise=true , address=145.79.227.14:4838 , piece_id=EELRNX7PWV4RCJNL2VM276BISNYGHTYDNOCSINOJQGYR2VYKZ3XA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.871587683 (514h1m13.742670718s) [noise=true , address=45.91.116.4:4940 , piece_id=ULDHQXBYQVAJGFQGPTMO6MGSEFRULSLP5KKB7TMBMBK55BOZBJCA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.872935639 (514h1m13.744018684s) [noise=true , address=145.79.227.41:4087 , piece_id=TAFBPAMWTHKFYAYYUNUY6N4J75AK7IVOLREEOGRXT2MOTDAUUQKQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.874270802 (514h1m13.745353847s) [noise=true , address=145.79.227.51:4410 , piece_id=ML5EDUBWPA6HUXPENG22JPT3N7L6CP6YMVF7VN4VF26KSDUMXFZQ, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.874771235 (514h1m13.74585428s) [noise=true , address=145.79.227.45:4229 , piece_id=YNST54V5BWHMJNJXDENSHC4L6USCKCJGJLAOGUJNTXTXKNAPU4LA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.876045380 (514h1m13.747128426s) [noise=true , address=145.79.227.50:4375 , piece_id=QXYI5OVTTGPXJAJA74JQKIS7NGCKFLJHZFZX25SM2KRPJWJKRHBA, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:13.879870645 (514h1m13.75095368s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Uploading piece... Dec 27 01:30:15.568895479 (514h1m15.439978524s) [upload=451052]: Next returned nil segment Dec 27 01:30:15.568912212 (514h1m15.439995257s) [upload=451052]: Waiting for error group managing segments... Dec 27 01:30:15.870087627 (514h1m15.741170662s) [noise=true , address=145.79.227.46:4602 , piece_id=JGGNTS27I6BF7YAE2DK2GC3QVGE6VOI3EETAGXHT3KTI63B7TLJA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.882024686 (514h1m15.753107721s) [noise=true , address=145.79.227.45:4229 , piece_id=YNST54V5BWHMJNJXDENSHC4L6USCKCJGJLAOGUJNTXTXKNAPU4LA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.888189796 (514h1m15.759272831s) [noise=true , address=145.79.227.5:4221 , piece_id=64X2B5A2SQNLSYXIR3RWWZFBS22MURKVKFMVDOIZ4YKAPC3GRPWQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.890313279 (514h1m15.761396314s) [noise=true , address=145.79.227.50:4375 , piece_id=QXYI5OVTTGPXJAJA74JQKIS7NGCKFLJHZFZX25SM2KRPJWJKRHBA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.913201270 (514h1m15.784284315s) [noise=true , address=145.79.227.48:4303 , piece_id=AGUTO6PWOVIJPDQ5X234CLSZOIG7QH4WLVTENFGWSG2NKGRJXKWA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.913495611 (514h1m15.784578656s) [noise=true , address=145.79.227.51:4410 , piece_id=ML5EDUBWPA6HUXPENG22JPT3N7L6CP6YMVF7VN4VF26KSDUMXFZQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:15.915460122 (514h1m15.786543167s) [noise=true , address=145.79.227.14:4838 , piece_id=EELRNX7PWV4RCJNL2VM276BISNYGHTYDNOCSINOJQGYR2VYKZ3XA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.044107473 (514h1m15.915190518s) [noise=true , address=77.90.31.10:4520 , piece_id=IKAL6ZPTSMXLOCRE7UGRZJJFMLQQRI447WSKY73IPAIYJQV4RFZQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.065944493 (514h1m15.937027538s) [noise=true , address=77.90.31.11:4556 , piece_id=WO47NGFXL44MBKVHJHJ7RYGOOLJFRA25V553R6JI2AWDVKVS4K2Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.097550454 (514h1m15.968633489s) [noise=true , address=45.91.116.7:4010 , piece_id=5SKXCDDP347KXNV36CYJDSP4FD52NL2NN6OLWINJRXC3EZWBCS2Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.131645786 (514h1m16.002728831s) [noise=true , address=45.91.116.5:4300 , piece_id=SUBGK6NTRDXARLBZDOHTI5ULKYNMWY3JQHMNE4ZPBQTFMD5VXEYA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.231608708 (514h1m16.102691753s) [noise=true , address=45.91.116.4:4940 , piece_id=ULDHQXBYQVAJGFQGPTMO6MGSEFRULSLP5KKB7TMBMBK55BOZBJCA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.304258012 (514h1m16.175341047s) [noise=true , address=145.79.227.41:4087 , piece_id=TAFBPAMWTHKFYAYYUNUY6N4J75AK7IVOLREEOGRXT2MOTDAUUQKQ, seg_pos={0 0}, upload=451052]: Done uploading piece. err: <nil> Dec 27 01:30:16.304277859 (514h1m16.175360894s) [seg_pos={0 0}, upload=451052]: Segment reached optimal threshold of 13 pieces. Dec 27 01:30:16.304495934 (514h1m16.175578969s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.6:4257: use of closed network connection Dec 27 01:30:16.304502487 (514h1m16.175585533s) [noise=true , address=145.79.227.6:4257 , piece_id=GX7PC7DKAXYIOHDYU2T6RTXRKCZCB2L6YH434UZ6VF4CXGND7D7Q, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304708971 (514h1m16.175792016s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 01:30:16.304716435 (514h1m16.17579947s) [noise=true , address=77.90.31.21:4877 , piece_id=24VFR7XB76ANIRHWNKYYJS4SWYZZXUKUYOEZSJEXBHGURKT5463A, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304867101 (514h1m16.175950156s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39510->45.91.116.13:4156: use of closed network connection Dec 27 01:30:16.304889835 (514h1m16.17597288s) [noise=true , address=45.91.116.13:4156 , piece_id=TK25RZNGHLJLED7GHM7XQPCUL4VYI235RR3VFQXEP7UO5MDPL5IA, seg_pos={0 0}, upload=451052]: Long tail cancellation due to optimal threshold Dec 27 01:30:16.304942595 (514h1m16.17602564s) [seg_pos={0 0}, upload=451052]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:30:16.305064067 (514h1m16.176147112s) [upload=451052]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:30:16.305071951 (514h1m16.176155007s) [upload=451052]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:30:16.589704007 (514h1m16.460787052s) [upload=451052]: Done uploading segments... Dec 27 01:30:16.589729295 (514h1m16.46081234s) [upload=451052]: Upload finished. err: <nil> Duration: 30.003256233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.XX] duration: 30.002477384s trace: https://jaeger.dev.storj.io/trace/682642501fea0fc8 FAIL (2025-12-27 01:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 (513h59m59.18294159s) [upload=451035]: Uploading segments... Dec 27 01:28:59.312107008 (513h59m59.183190043s) [upload=451035]: Got next segment. Inline: false Dec 27 01:28:59.312159597 (513h59m59.183242642s) [upload=451035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:28:59.312166451 (513h59m59.183249486s) [upload=451035]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:28:59.559635081 (513h59m59.430718126s) [seg_pos={0 0}, upload=451035]: Begin upload segment... Dec 27 01:28:59.559674847 (513h59m59.430757892s) [seg_pos={0 0}, upload=451035]: Done begin upload segment. Dec 27 01:28:59.559704513 (513h59m59.430787558s) [noise=true , address=145.79.227.36:4518 , piece_id=XSLDIZWOROEP2VH4ZAHK6Y5GRBNLUT2IS6QAQWJKL7P5UNHOS4ZQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.559771020 (513h59m59.430854055s) [noise=true , address=145.79.227.11:4732 , piece_id=RTPK6B4XUOHZFYOQKOD7CZOY6MTLEQIMZMMIUOB4FL57K4BWMISQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.560378136 (513h59m59.431461171s) [noise=true , address=145.79.227.39:4014 , piece_id=UQIREQYUVDYGVH5D4AS7SYGPHS4BKCPLBLXDE7OXWOKEHNZ6D6HA, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.560878689 (513h59m59.431961745s) [noise=true , address=145.79.227.42:4121 , piece_id=YUCFQQ5534A3OWBGC4EKWI2LF57HMN2CVP2PW7JDVZQG6MOYJGWA, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.561204099 (513h59m59.432287144s) [noise=true , address=145.79.227.13:4804 , piece_id=7DG72SFYJRZASAFQAMHR6G4JATC3RRERRQNMNJVE6PALR7U4TFQQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.561693210 (513h59m59.432776245s) [noise=true , address=145.79.227.17:4971 , piece_id=NUAIWUZ6FRGQR6UGMSA5JWT2WYU2SYCMQWV2KUX3KPQ65FSQGGCQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.562038117 (513h59m59.433121162s) [noise=true , address=45.91.116.12:4227 , piece_id=2SVX6ERRV6TNJBQUPBFXZDANQTWGTHNEDFCJGRGS2T6RNEGU42SQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.562160180 (513h59m59.433243215s) [noise=true , address=45.91.116.13:4154 , piece_id=XPGGZP4AB2IM4SFH5NJDFZXNLNKMHPN3R2BCP6E5MJG7BCPYSJ2Q, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.562496239 (513h59m59.433579274s) [noise=true , address=45.91.116.6:4975 , piece_id=RENL2BCWV67XDYK7CTUMG5U5VBANBWPIJHKR4UQ4U4BOLNQJGUCA, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.562640333 (513h59m59.433723368s) [noise=true , address=77.90.31.10:4517 , piece_id=ZPLV3IBZKOFSMAUJMDTYHTBPXI4OLSZUB2N3PXJRFODG3S2VCU6A, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.562820045 (513h59m59.43390308s) [noise=true , address=145.79.227.38:4977 , piece_id=QEQ5ELWOGGPYH6PI47N5VOM3VPHBWZNDT6LQVFZ66XBFYFD4AA4Q, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.563070052 (513h59m59.434153107s) [noise=true , address=145.79.227.15:4875 , piece_id=2VHWHYQNOWRHYNDJ5BRZA6RIW6PQHEHVTX3GM7WR6K7VAJG7R2TQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.563125708 (513h59m59.434208753s) [noise=true , address=145.79.227.51:4410 , piece_id=AK6JFNUJBJRCSG5MGNOAFIA2L22U5MNJ3SNRISAAIWHXDLQK2OXQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.563488980 (513h59m59.434572015s) [noise=true , address=145.79.227.34:4905 , piece_id=BNM6Y7WU2O6RW422NX7UJJFQRZ223RE3X6TASJ6CCBIELYYS74KQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.563960778 (513h59m59.435043813s) [noise=true , address=45.91.116.8:4048 , piece_id=EJRHOS4DUCLY6M6AAGZPZELBSM6Z5XBB5JNP357VEZJIKC5QCQYQ, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:28:59.565972398 (513h59m59.437055444s) [noise=true , address=45.91.116.11:4118 , piece_id=UDF3O2GYEH5RUFO7G7YNAN6QKWVE6HV3ZJOAISXKLFTPZXMWHKLA, seg_pos={0 0}, upload=451035]: Uploading piece... Dec 27 01:29:01.178702343 (514h0m1.049785388s) [upload=451035]: Next returned nil segment Dec 27 01:29:01.178715338 (514h0m1.049798383s) [upload=451035]: Waiting for error group managing segments... Dec 27 01:29:01.562907088 (514h0m1.433990133s) [noise=true , address=145.79.227.13:4804 , piece_id=7DG72SFYJRZASAFQAMHR6G4JATC3RRERRQNMNJVE6PALR7U4TFQQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.565529391 (514h0m1.436612436s) [noise=true , address=145.79.227.11:4732 , piece_id=RTPK6B4XUOHZFYOQKOD7CZOY6MTLEQIMZMMIUOB4FL57K4BWMISQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.568843021 (514h0m1.439926066s) [noise=true , address=145.79.227.42:4121 , piece_id=YUCFQQ5534A3OWBGC4EKWI2LF57HMN2CVP2PW7JDVZQG6MOYJGWA, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.571371665 (514h0m1.44245471s) [noise=true , address=145.79.227.51:4410 , piece_id=AK6JFNUJBJRCSG5MGNOAFIA2L22U5MNJ3SNRISAAIWHXDLQK2OXQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.572611375 (514h0m1.44369442s) [noise=true , address=145.79.227.38:4977 , piece_id=QEQ5ELWOGGPYH6PI47N5VOM3VPHBWZNDT6LQVFZ66XBFYFD4AA4Q, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.593166556 (514h0m1.464249601s) [noise=true , address=145.79.227.15:4875 , piece_id=2VHWHYQNOWRHYNDJ5BRZA6RIW6PQHEHVTX3GM7WR6K7VAJG7R2TQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.596547063 (514h0m1.467630108s) [noise=true , address=145.79.227.36:4518 , piece_id=XSLDIZWOROEP2VH4ZAHK6Y5GRBNLUT2IS6QAQWJKL7P5UNHOS4ZQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.600795171 (514h0m1.471878206s) [noise=true , address=145.79.227.39:4014 , piece_id=UQIREQYUVDYGVH5D4AS7SYGPHS4BKCPLBLXDE7OXWOKEHNZ6D6HA, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.603180022 (514h0m1.474263067s) [noise=true , address=145.79.227.17:4971 , piece_id=NUAIWUZ6FRGQR6UGMSA5JWT2WYU2SYCMQWV2KUX3KPQ65FSQGGCQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.787781607 (514h0m1.658864662s) [noise=true , address=45.91.116.13:4154 , piece_id=XPGGZP4AB2IM4SFH5NJDFZXNLNKMHPN3R2BCP6E5MJG7BCPYSJ2Q, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.814869906 (514h0m1.685953132s) [noise=true , address=45.91.116.6:4975 , piece_id=RENL2BCWV67XDYK7CTUMG5U5VBANBWPIJHKR4UQ4U4BOLNQJGUCA, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.884729978 (514h0m1.755813033s) [noise=true , address=45.91.116.8:4048 , piece_id=EJRHOS4DUCLY6M6AAGZPZELBSM6Z5XBB5JNP357VEZJIKC5QCQYQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.914185245 (514h0m1.7852683s) [noise=true , address=45.91.116.11:4118 , piece_id=UDF3O2GYEH5RUFO7G7YNAN6QKWVE6HV3ZJOAISXKLFTPZXMWHKLA, seg_pos={0 0}, upload=451035]: Done uploading piece. err: <nil> Dec 27 01:29:01.914214020 (514h0m1.785297065s) [seg_pos={0 0}, upload=451035]: Segment reached optimal threshold of 13 pieces. Dec 27 01:29:01.914586129 (514h0m1.785669184s) [noise=true , address=77.90.31.10:4517 , piece_id=ZPLV3IBZKOFSMAUJMDTYHTBPXI4OLSZUB2N3PXJRFODG3S2VCU6A, seg_pos={0 0}, upload=451035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:52098->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:52098->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52098->77.90.31.10:4517: use of closed network connection Dec 27 01:29:01.914596488 (514h0m1.785679533s) [noise=true , address=77.90.31.10:4517 , piece_id=ZPLV3IBZKOFSMAUJMDTYHTBPXI4OLSZUB2N3PXJRFODG3S2VCU6A, seg_pos={0 0}, upload=451035]: Long tail cancellation due to optimal threshold Dec 27 01:29:01.914842887 (514h0m1.785925932s) [noise=true , address=45.91.116.12:4227 , piece_id=2SVX6ERRV6TNJBQUPBFXZDANQTWGTHNEDFCJGRGS2T6RNEGU42SQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:53056->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:53056->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53056->45.91.116.12:4227: use of closed network connection Dec 27 01:29:01.914857094 (514h0m1.785940139s) [noise=true , address=45.91.116.12:4227 , piece_id=2SVX6ERRV6TNJBQUPBFXZDANQTWGTHNEDFCJGRGS2T6RNEGU42SQ, seg_pos={0 0}, upload=451035]: Long tail cancellation due to optimal threshold Dec 27 01:29:01.915053057 (514h0m1.786136112s) [noise=true , address=145.79.227.34:4905 , piece_id=BNM6Y7WU2O6RW422NX7UJJFQRZ223RE3X6TASJ6CCBIELYYS74KQ, seg_pos={0 0}, upload=451035]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 01:29:01.915071272 (514h0m1.786154308s) [noise=true , address=145.79.227.34:4905 , piece_id=BNM6Y7WU2O6RW422NX7UJJFQRZ223RE3X6TASJ6CCBIELYYS74KQ, seg_pos={0 0}, upload=451035]: Long tail cancellation due to optimal threshold Dec 27 01:29:01.915121508 (514h0m1.786204553s) [seg_pos={0 0}, upload=451035]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:29:01.915234222 (514h0m1.786317267s) [upload=451035]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:29:01.915242328 (514h0m1.786325393s) [upload=451035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:29:02.215485568 (514h0m2.086568613s) [upload=451035]: Done uploading segments... Dec 27 01:29:02.215508873 (514h0m2.086591918s) [upload=451035]: Upload finished. err: <nil> Duration: 30.0024857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.X] duration: 30.003405252s trace: https://jaeger.dev.storj.io/trace/3ece8a0947689a12 FAIL (2025-12-27 01:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 72VZL7N6G3242IBH3CERNFMLCDD5CZJNYR5LGWDWCT577TKW5U3A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: DP42RZBR5JPCNHN6QXEP2TL62SRHYQYPOOILB4GQKDA2S2DD5E6Q) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: QLZP3YJ4SVKDDONMHVP3LIQZ5BNGIRH5ST3DL5WSXPFIKJRVSPEQ) context deadline exceeded ======LOGS====== Dec 27 01:28:01.305836228 (513h59m1.176919263s) [upload=451021]: Starting upload Dec 27 01:28:01.305934735 (513h59m1.17701777s) [upload=451021]: Done starting upload Dec 27 01:28:01.306049003 (513h59m1.177132048s) [upload=451021]: Uploading segments... Dec 27 01:28:01.306238163 (513h59m1.177321208s) [upload=451021]: Got next segment. Inline: false Dec 27 01:28:01.306293970 (513h59m1.177377005s) [upload=451021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:28:01.306300272 (513h59m1.177383307s) [upload=451021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:28:01.550287213 (513h59m1.421370258s) [seg_pos={0 0}, upload=451021]: Begin upload segment... Dec 27 01:28:01.550339423 (513h59m1.421422468s) [seg_pos={0 0}, upload=451021]: Done begin upload segment. Dec 27 01:28:01.550368619 (513h59m1.421451664s) [noise=true , address=45.91.116.10:4082 , piece_id=DP42RZBR5JPCNHN6QXEP2TL62SRHYQYPOOILB4GQKDA2S2DD5E6Q, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.550379950 (513h59m1.421462985s) [noise=true , address=145.79.227.47:4267 , piece_id=WVFOW6JJC5MJCYGTY6WUB43XPUUMUEND2734N3PN7C4HGQRLQUTA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.550920649 (513h59m1.422003684s) [noise=true , address=145.79.227.13:4804 , piece_id=MIS3VXJUPMDYILTVW2IDV45R2YVYVNS5N7QKOSG2XSMRRQ7GJDGA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.551240749 (513h59m1.422323795s) [noise=true , address=45.91.116.8:4048 , piece_id=VTDDU6WJRHOIIWBOS3SGX6HJ26EU7AGONVTZFXTLUBGPT6HXK2FA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.551304730 (513h59m1.422387776s) [noise=true , address=145.79.227.12:4765 , piece_id=RFASCTM6H4Z6XE7MYNAWXX4HEZ2PPMUEGSYGR3RKMIB4L5GLRF3A, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.552451705 (513h59m1.42353475s) [noise=true , address=45.91.116.11:4118 , piece_id=QLZP3YJ4SVKDDONMHVP3LIQZ5BNGIRH5ST3DL5WSXPFIKJRVSPEQ, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.552838892 (513h59m1.423921927s) [noise=true , address=145.79.227.11:4731 , piece_id=RW7CBCTZHC2WZODNVKPLOSXP3Y64GHJ6K7STVQFES3KN2RX7IX4Q, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.553395371 (513h59m1.424478406s) [noise=true , address=145.79.227.49:4337 , piece_id=6EUISJWQ3V5DWDWHDNBEZ44FHTMLBM7FJ7IU6QTDZV4UCQNDWEBA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.553911124 (513h59m1.424994159s) [noise=true , address=145.79.227.14:4837 , piece_id=74BVUBVWPZG7HV33L7N6H5V4FOZCBQKYWRJOTIBQKQVNVRZWHHEA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.554469226 (513h59m1.425552291s) [noise=true , address=145.79.227.6:4258 , piece_id=7MBQL37ZFRIXLOLTCVRIFNBDTSOH4RX5DI5CBLF3K5RIFI2SJIDQ, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.555138180 (513h59m1.426221225s) [noise=true , address=145.79.227.51:4409 , piece_id=UTOZENLEMVYBW46QVVORRJ7FOV44K4IUZEQFWLJNPINQJ6KE6GCA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.556528868 (513h59m1.427611913s) [noise=true , address=77.90.31.19:4844 , piece_id=3MJCKLMKIXMEXMLH36K5BWVTNHKBXZBRRXG5B7423AZE7GOWFDAA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.559574698 (513h59m1.430657743s) [noise=true , address=77.90.31.18:4808 , piece_id=72VZL7N6G3242IBH3CERNFMLCDD5CZJNYR5LGWDWCT577TKW5U3A, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.568430621 (513h59m1.439513666s) [noise=true , address=45.91.116.12:4226 , piece_id=XULN74UOOQ522ZJKHV53O3O6GFQYQFZH3KK7WR4YHFCDIKQJOMWQ, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.574960505 (513h59m1.44604356s) [noise=true , address=45.91.116.9:4263 , piece_id=VWOFTEPNMJ3B4HAK6CLD3N6ZNL2GF3GM5VL4VLDCM5Z2Q5BAJQHQ, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:01.575762021 (513h59m1.446845066s) [noise=true , address=145.79.227.42:4121 , piece_id=X4GSGLA7VQW2ONE2FQEEKJGTGPQFV72D7TXIKIYLTDGQWVCKPAKA, seg_pos={0 0}, upload=451021]: Uploading piece... Dec 27 01:28:03.182205822 (513h59m3.053288857s) [upload=451021]: Next returned nil segment Dec 27 01:28:03.182222624 (513h59m3.053305669s) [upload=451021]: Waiting for error group managing segments... Dec 27 01:28:03.567725941 (513h59m3.438808987s) [noise=true , address=145.79.227.13:4804 , piece_id=MIS3VXJUPMDYILTVW2IDV45R2YVYVNS5N7QKOSG2XSMRRQ7GJDGA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.575365448 (513h59m3.446448493s) [noise=true , address=145.79.227.6:4258 , piece_id=7MBQL37ZFRIXLOLTCVRIFNBDTSOH4RX5DI5CBLF3K5RIFI2SJIDQ, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.578024731 (513h59m3.449107766s) [noise=true , address=145.79.227.11:4731 , piece_id=RW7CBCTZHC2WZODNVKPLOSXP3Y64GHJ6K7STVQFES3KN2RX7IX4Q, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.580414501 (513h59m3.451497536s) [noise=true , address=145.79.227.14:4837 , piece_id=74BVUBVWPZG7HV33L7N6H5V4FOZCBQKYWRJOTIBQKQVNVRZWHHEA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.583409314 (513h59m3.454492349s) [noise=true , address=145.79.227.12:4765 , piece_id=RFASCTM6H4Z6XE7MYNAWXX4HEZ2PPMUEGSYGR3RKMIB4L5GLRF3A, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.583861635 (513h59m3.45494468s) [noise=true , address=145.79.227.51:4409 , piece_id=UTOZENLEMVYBW46QVVORRJ7FOV44K4IUZEQFWLJNPINQJ6KE6GCA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.588774089 (513h59m3.459857134s) [noise=true , address=145.79.227.49:4337 , piece_id=6EUISJWQ3V5DWDWHDNBEZ44FHTMLBM7FJ7IU6QTDZV4UCQNDWEBA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.594125970 (513h59m3.465209015s) [noise=true , address=145.79.227.42:4121 , piece_id=X4GSGLA7VQW2ONE2FQEEKJGTGPQFV72D7TXIKIYLTDGQWVCKPAKA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.603096882 (513h59m3.474179927s) [noise=true , address=145.79.227.47:4267 , piece_id=WVFOW6JJC5MJCYGTY6WUB43XPUUMUEND2734N3PN7C4HGQRLQUTA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.756707195 (513h59m3.62779024s) [noise=true , address=77.90.31.18:4808 , piece_id=72VZL7N6G3242IBH3CERNFMLCDD5CZJNYR5LGWDWCT577TKW5U3A, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.765150672 (513h59m3.636233717s) [noise=true , address=45.91.116.10:4082 , piece_id=DP42RZBR5JPCNHN6QXEP2TL62SRHYQYPOOILB4GQKDA2S2DD5E6Q, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.805147011 (513h59m3.676230056s) [noise=true , address=45.91.116.12:4226 , piece_id=XULN74UOOQ522ZJKHV53O3O6GFQYQFZH3KK7WR4YHFCDIKQJOMWQ, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.844157250 (513h59m3.715240296s) [noise=true , address=45.91.116.11:4118 , piece_id=QLZP3YJ4SVKDDONMHVP3LIQZ5BNGIRH5ST3DL5WSXPFIKJRVSPEQ, seg_pos={0 0}, upload=451021]: Done uploading piece. err: <nil> Dec 27 01:28:03.844183200 (513h59m3.715266245s) [seg_pos={0 0}, upload=451021]: Segment reached optimal threshold of 13 pieces. Dec 27 01:28:03.844434448 (513h59m3.715517483s) [noise=true , address=45.91.116.9:4263 , piece_id=VWOFTEPNMJ3B4HAK6CLD3N6ZNL2GF3GM5VL4VLDCM5Z2Q5BAJQHQ, seg_pos={0 0}, upload=451021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 01:28:03.844442324 (513h59m3.715525369s) [noise=true , address=45.91.116.9:4263 , piece_id=VWOFTEPNMJ3B4HAK6CLD3N6ZNL2GF3GM5VL4VLDCM5Z2Q5BAJQHQ, seg_pos={0 0}, upload=451021]: Long tail cancellation due to optimal threshold Dec 27 01:28:03.844625553 (513h59m3.715708598s) [noise=true , address=77.90.31.19:4844 , piece_id=3MJCKLMKIXMEXMLH36K5BWVTNHKBXZBRRXG5B7423AZE7GOWFDAA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 01:28:03.844644188 (513h59m3.715727223s) [noise=true , address=77.90.31.19:4844 , piece_id=3MJCKLMKIXMEXMLH36K5BWVTNHKBXZBRRXG5B7423AZE7GOWFDAA, seg_pos={0 0}, upload=451021]: Long tail cancellation due to optimal threshold Dec 27 01:28:03.844735803 (513h59m3.715818848s) [noise=true , address=45.91.116.8:4048 , piece_id=VTDDU6WJRHOIIWBOS3SGX6HJ26EU7AGONVTZFXTLUBGPT6HXK2FA, seg_pos={0 0}, upload=451021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 01:28:03.844744930 (513h59m3.715827975s) [noise=true , address=45.91.116.8:4048 , piece_id=VTDDU6WJRHOIIWBOS3SGX6HJ26EU7AGONVTZFXTLUBGPT6HXK2FA, seg_pos={0 0}, upload=451021]: Long tail cancellation due to optimal threshold Dec 27 01:28:03.844765920 (513h59m3.715848965s) [seg_pos={0 0}, upload=451021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:28:03.844864137 (513h59m3.715947182s) [upload=451021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:28:03.844870479 (513h59m3.715953524s) [upload=451021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:28:04.141096471 (513h59m4.012179516s) [upload=451021]: Done uploading segments... Dec 27 01:28:04.141122220 (513h59m4.012205265s) [upload=451021]: Upload finished. err: <nil> Duration: 30.003429278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X.] duration: 6.053423949s trace: https://jaeger.dev.storj.io/trace/4517b0c61ae54dc FAIL (2025-12-27 01:27:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:26:21.084987333 (513h57m20.956070368s) [upload=450996]: Starting upload Dec 27 01:26:21.085041957 (513h57m20.956124992s) [upload=450996]: Done starting upload Dec 27 01:26:21.085135424 (513h57m20.956218459s) [upload=450996]: Uploading segments... Dec 27 01:26:21.085234353 (513h57m20.956317388s) [upload=450996]: Got next segment. Inline: false Dec 27 01:26:21.085270692 (513h57m20.956353727s) [upload=450996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:26:21.085277625 (513h57m20.95636067s) [upload=450996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:26:21.331850341 (513h57m21.202933386s) [seg_pos={0 0}, upload=450996]: Begin upload segment... Dec 27 01:26:21.331903873 (513h57m21.202986918s) [seg_pos={0 0}, upload=450996]: Done begin upload segment. Dec 27 01:26:21.331932698 (513h57m21.203015763s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.331945783 (513h57m21.203028818s) [noise=true , address=145.79.227.11:4732 , piece_id=ZJUZF5IM542C5NQA27XWFS4QGEGZWYS6YQVYJNJBDVOJTNV3Z4DQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332012380 (513h57m21.203095425s) [noise=true , address=145.79.227.47:4266 , piece_id=RGFZKPDL6KZVHWSDX65KIKRWPCW3LUWHLPX2PKOX4WYZDC3XK4KA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332453290 (513h57m21.203536335s) [noise=true , address=77.90.31.19:4841 , piece_id=JJODUMVZNODM6OT2NYEFUVP5HZ6WMTTYPLI4WLDKKMOXEBTFMPEA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332868060 (513h57m21.203951095s) [noise=true , address=145.79.227.15:4874 , piece_id=H6CJJIIFNMYGPD74CIIASNE2YCPVFJK4U3NWDOD6RERGCICWKOQQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.333456190 (513h57m21.204539225s) [noise=true , address=45.91.116.12:4226 , piece_id=5O4CBUMTH4MS7GVXN6RU4P3IS5P2O3VMY6QIKPRMMR3TC5HBCHRQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334067723 (513h57m21.205150779s) [noise=true , address=145.79.227.36:4517 , piece_id=XEONHOXZ7UE7M67CPXTLCTJWGOZ5OGAXCTWOJLSP5RSLQXJXTD2Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334602041 (513h57m21.205685086s) [noise=true , address=145.79.227.17:4972 , piece_id=DHWB6437KH4IMLZLABCFT3IDEDNCZR7EVRHT6H5R2EFE7MRHZJPQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334834013 (513h57m21.205917048s) [noise=true , address=145.79.227.13:4804 , piece_id=7CMU55UJMN6KCIWRBM34SSTWMAPYS5EQ26XEFQS3Y2LUXF2YPYUQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335052488 (513h57m21.206135533s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335436299 (513h57m21.206519344s) [noise=true , address=45.91.116.7:4011 , piece_id=AECBQKNXIAJUXYSV33TDMHALYBVMIQKHFCMZRPEHL3PZW562HTYA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335499981 (513h57m21.206583026s) [noise=true , address=145.79.227.38:4978 , piece_id=OEU52KR6WDZPQNTUTUKBOP4VCE34ZRTG56FPEIQ2QCP5IHYPUFSQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335942143 (513h57m21.207025188s) [noise=true , address=145.79.227.2:4113 , piece_id=W4JSUIZC2S2DCGZZBZNDZSYV6YSEUXXWUHGI4JOPFGKKWUAO7X4Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335982540 (513h57m21.207065575s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.336553507 (513h57m21.207636542s) [noise=true , address=77.90.31.17:4770 , piece_id=G3O2W7YSQAVLZQAF7KB64FMFSTAT3Y6WJM6T64SU4JIO4L65QT2Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.337117371 (513h57m21.208200406s) [noise=true , address=145.79.227.39:4015 , piece_id=JL37LS6T7VFI5OUMKJD6NIER5ZOUAI6WBIRKMZGHINIT47A64FVA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:22.954935408 (513h57m22.826018453s) [upload=450996]: Next returned nil segment Dec 27 01:26:22.954947952 (513h57m22.826030987s) [upload=450996]: Waiting for error group managing segments... Dec 27 01:26:23.340856919 (513h57m23.211939954s) [noise=true , address=145.79.227.11:4732 , piece_id=ZJUZF5IM542C5NQA27XWFS4QGEGZWYS6YQVYJNJBDVOJTNV3Z4DQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.342572405 (513h57m23.213655441s) [noise=true , address=145.79.227.39:4015 , piece_id=JL37LS6T7VFI5OUMKJD6NIER5ZOUAI6WBIRKMZGHINIT47A64FVA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.352767227 (513h57m23.223850272s) [noise=true , address=145.79.227.13:4804 , piece_id=7CMU55UJMN6KCIWRBM34SSTWMAPYS5EQ26XEFQS3Y2LUXF2YPYUQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.362162518 (513h57m23.233245553s) [noise=true , address=145.79.227.2:4113 , piece_id=W4JSUIZC2S2DCGZZBZNDZSYV6YSEUXXWUHGI4JOPFGKKWUAO7X4Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.365660548 (513h57m23.236743583s) [noise=true , address=145.79.227.47:4266 , piece_id=RGFZKPDL6KZVHWSDX65KIKRWPCW3LUWHLPX2PKOX4WYZDC3XK4KA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.366241184 (513h57m23.237324219s) [noise=true , address=145.79.227.38:4978 , piece_id=OEU52KR6WDZPQNTUTUKBOP4VCE34ZRTG56FPEIQ2QCP5IHYPUFSQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.367320028 (513h57m23.238403073s) [noise=true , address=145.79.227.15:4874 , piece_id=H6CJJIIFNMYGPD74CIIASNE2YCPVFJK4U3NWDOD6RERGCICWKOQQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.368284885 (513h57m23.23936793s) [noise=true , address=145.79.227.36:4517 , piece_id=XEONHOXZ7UE7M67CPXTLCTJWGOZ5OGAXCTWOJLSP5RSLQXJXTD2Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.375328888 (513h57m23.246411924s) [noise=true , address=145.79.227.17:4972 , piece_id=DHWB6437KH4IMLZLABCFT3IDEDNCZR7EVRHT6H5R2EFE7MRHZJPQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.527522074 (513h57m23.398605109s) [noise=true , address=77.90.31.17:4770 , piece_id=G3O2W7YSQAVLZQAF7KB64FMFSTAT3Y6WJM6T64SU4JIO4L65QT2Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.551436219 (513h57m23.422519264s) [noise=true , address=45.91.116.7:4011 , piece_id=AECBQKNXIAJUXYSV33TDMHALYBVMIQKHFCMZRPEHL3PZW562HTYA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.621950626 (513h57m23.493033661s) [noise=true , address=77.90.31.19:4841 , piece_id=JJODUMVZNODM6OT2NYEFUVP5HZ6WMTTYPLI4WLDKKMOXEBTFMPEA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.688101075 (513h57m23.55918412s) [noise=true , address=45.91.116.12:4226 , piece_id=5O4CBUMTH4MS7GVXN6RU4P3IS5P2O3VMY6QIKPRMMR3TC5HBCHRQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.688127215 (513h57m23.55921025s) [seg_pos={0 0}, upload=450996]: Segment reached optimal threshold of 13 pieces. Dec 27 01:26:23.688362423 (513h57m23.559445468s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 01:26:23.688372322 (513h57m23.559455377s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688561652 (513h57m23.559644697s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 01:26:23.688570579 (513h57m23.559653614s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688887593 (513h57m23.559970648s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:56024->45.91.116.4:4940: use of closed network connection; context canceled Dec 27 01:26:23.688909223 (513h57m23.559992258s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688942136 (513h57m23.560025171s) [seg_pos={0 0}, upload=450996]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:26:23.689045112 (513h57m23.560128147s) [upload=450996]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:26:23.689052376 (513h57m23.560135411s) [upload=450996]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:26:23.986832156 (513h57m23.857915191s) [upload=450996]: Done uploading segments... Dec 27 01:26:23.986849720 (513h57m23.857932765s) [upload=450996]: Upload finished. err: <nil> Duration: 6.053449317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.X] duration: 30.0030342s trace: https://jaeger.dev.storj.io/trace/4517b0c61ae54dc FAIL (2025-12-27 01:26:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:26:21.084987333 (513h57m20.956070368s) [upload=450996]: Starting upload Dec 27 01:26:21.085041957 (513h57m20.956124992s) [upload=450996]: Done starting upload Dec 27 01:26:21.085135424 (513h57m20.956218459s) [upload=450996]: Uploading segments... Dec 27 01:26:21.085234353 (513h57m20.956317388s) [upload=450996]: Got next segment. Inline: false Dec 27 01:26:21.085270692 (513h57m20.956353727s) [upload=450996]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:26:21.085277625 (513h57m20.95636067s) [upload=450996]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:26:21.331850341 (513h57m21.202933386s) [seg_pos={0 0}, upload=450996]: Begin upload segment... Dec 27 01:26:21.331903873 (513h57m21.202986918s) [seg_pos={0 0}, upload=450996]: Done begin upload segment. Dec 27 01:26:21.331932698 (513h57m21.203015763s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.331945783 (513h57m21.203028818s) [noise=true , address=145.79.227.11:4732 , piece_id=ZJUZF5IM542C5NQA27XWFS4QGEGZWYS6YQVYJNJBDVOJTNV3Z4DQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332012380 (513h57m21.203095425s) [noise=true , address=145.79.227.47:4266 , piece_id=RGFZKPDL6KZVHWSDX65KIKRWPCW3LUWHLPX2PKOX4WYZDC3XK4KA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332453290 (513h57m21.203536335s) [noise=true , address=77.90.31.19:4841 , piece_id=JJODUMVZNODM6OT2NYEFUVP5HZ6WMTTYPLI4WLDKKMOXEBTFMPEA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.332868060 (513h57m21.203951095s) [noise=true , address=145.79.227.15:4874 , piece_id=H6CJJIIFNMYGPD74CIIASNE2YCPVFJK4U3NWDOD6RERGCICWKOQQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.333456190 (513h57m21.204539225s) [noise=true , address=45.91.116.12:4226 , piece_id=5O4CBUMTH4MS7GVXN6RU4P3IS5P2O3VMY6QIKPRMMR3TC5HBCHRQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334067723 (513h57m21.205150779s) [noise=true , address=145.79.227.36:4517 , piece_id=XEONHOXZ7UE7M67CPXTLCTJWGOZ5OGAXCTWOJLSP5RSLQXJXTD2Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334602041 (513h57m21.205685086s) [noise=true , address=145.79.227.17:4972 , piece_id=DHWB6437KH4IMLZLABCFT3IDEDNCZR7EVRHT6H5R2EFE7MRHZJPQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.334834013 (513h57m21.205917048s) [noise=true , address=145.79.227.13:4804 , piece_id=7CMU55UJMN6KCIWRBM34SSTWMAPYS5EQ26XEFQS3Y2LUXF2YPYUQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335052488 (513h57m21.206135533s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335436299 (513h57m21.206519344s) [noise=true , address=45.91.116.7:4011 , piece_id=AECBQKNXIAJUXYSV33TDMHALYBVMIQKHFCMZRPEHL3PZW562HTYA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335499981 (513h57m21.206583026s) [noise=true , address=145.79.227.38:4978 , piece_id=OEU52KR6WDZPQNTUTUKBOP4VCE34ZRTG56FPEIQ2QCP5IHYPUFSQ, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335942143 (513h57m21.207025188s) [noise=true , address=145.79.227.2:4113 , piece_id=W4JSUIZC2S2DCGZZBZNDZSYV6YSEUXXWUHGI4JOPFGKKWUAO7X4Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.335982540 (513h57m21.207065575s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.336553507 (513h57m21.207636542s) [noise=true , address=77.90.31.17:4770 , piece_id=G3O2W7YSQAVLZQAF7KB64FMFSTAT3Y6WJM6T64SU4JIO4L65QT2Q, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:21.337117371 (513h57m21.208200406s) [noise=true , address=145.79.227.39:4015 , piece_id=JL37LS6T7VFI5OUMKJD6NIER5ZOUAI6WBIRKMZGHINIT47A64FVA, seg_pos={0 0}, upload=450996]: Uploading piece... Dec 27 01:26:22.954935408 (513h57m22.826018453s) [upload=450996]: Next returned nil segment Dec 27 01:26:22.954947952 (513h57m22.826030987s) [upload=450996]: Waiting for error group managing segments... Dec 27 01:26:23.340856919 (513h57m23.211939954s) [noise=true , address=145.79.227.11:4732 , piece_id=ZJUZF5IM542C5NQA27XWFS4QGEGZWYS6YQVYJNJBDVOJTNV3Z4DQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.342572405 (513h57m23.213655441s) [noise=true , address=145.79.227.39:4015 , piece_id=JL37LS6T7VFI5OUMKJD6NIER5ZOUAI6WBIRKMZGHINIT47A64FVA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.352767227 (513h57m23.223850272s) [noise=true , address=145.79.227.13:4804 , piece_id=7CMU55UJMN6KCIWRBM34SSTWMAPYS5EQ26XEFQS3Y2LUXF2YPYUQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.362162518 (513h57m23.233245553s) [noise=true , address=145.79.227.2:4113 , piece_id=W4JSUIZC2S2DCGZZBZNDZSYV6YSEUXXWUHGI4JOPFGKKWUAO7X4Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.365660548 (513h57m23.236743583s) [noise=true , address=145.79.227.47:4266 , piece_id=RGFZKPDL6KZVHWSDX65KIKRWPCW3LUWHLPX2PKOX4WYZDC3XK4KA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.366241184 (513h57m23.237324219s) [noise=true , address=145.79.227.38:4978 , piece_id=OEU52KR6WDZPQNTUTUKBOP4VCE34ZRTG56FPEIQ2QCP5IHYPUFSQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.367320028 (513h57m23.238403073s) [noise=true , address=145.79.227.15:4874 , piece_id=H6CJJIIFNMYGPD74CIIASNE2YCPVFJK4U3NWDOD6RERGCICWKOQQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.368284885 (513h57m23.23936793s) [noise=true , address=145.79.227.36:4517 , piece_id=XEONHOXZ7UE7M67CPXTLCTJWGOZ5OGAXCTWOJLSP5RSLQXJXTD2Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.375328888 (513h57m23.246411924s) [noise=true , address=145.79.227.17:4972 , piece_id=DHWB6437KH4IMLZLABCFT3IDEDNCZR7EVRHT6H5R2EFE7MRHZJPQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.527522074 (513h57m23.398605109s) [noise=true , address=77.90.31.17:4770 , piece_id=G3O2W7YSQAVLZQAF7KB64FMFSTAT3Y6WJM6T64SU4JIO4L65QT2Q, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.551436219 (513h57m23.422519264s) [noise=true , address=45.91.116.7:4011 , piece_id=AECBQKNXIAJUXYSV33TDMHALYBVMIQKHFCMZRPEHL3PZW562HTYA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.621950626 (513h57m23.493033661s) [noise=true , address=77.90.31.19:4841 , piece_id=JJODUMVZNODM6OT2NYEFUVP5HZ6WMTTYPLI4WLDKKMOXEBTFMPEA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.688101075 (513h57m23.55918412s) [noise=true , address=45.91.116.12:4226 , piece_id=5O4CBUMTH4MS7GVXN6RU4P3IS5P2O3VMY6QIKPRMMR3TC5HBCHRQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: <nil> Dec 27 01:26:23.688127215 (513h57m23.55921025s) [seg_pos={0 0}, upload=450996]: Segment reached optimal threshold of 13 pieces. Dec 27 01:26:23.688362423 (513h57m23.559445468s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 01:26:23.688372322 (513h57m23.559455377s) [noise=true , address=45.91.116.13:4154 , piece_id=26DS6A6TZDXRIZKDYDKZMGJURT3DNXY77KNVWIOYQLYUZCNQJ5QQ, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688561652 (513h57m23.559644697s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 01:26:23.688570579 (513h57m23.559653614s) [noise=true , address=77.90.31.10:4519 , piece_id=XOR3S5DUEAWA2G3CDVJ243VYJASJVZTVQNSN7NCLGMAQJMBXNVAA, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688887593 (513h57m23.559970648s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:56024->45.91.116.4:4940: use of closed network connection; context canceled Dec 27 01:26:23.688909223 (513h57m23.559992258s) [noise=true , address=45.91.116.4:4940 , piece_id=6TN72HZIUKWEI7PZDXOVZJJ7B4IOFG2K7GULFIOC6WRD3KSU55DA, seg_pos={0 0}, upload=450996]: Long tail cancellation due to optimal threshold Dec 27 01:26:23.688942136 (513h57m23.560025171s) [seg_pos={0 0}, upload=450996]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:26:23.689045112 (513h57m23.560128147s) [upload=450996]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:26:23.689052376 (513h57m23.560135411s) [upload=450996]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:26:23.986832156 (513h57m23.857915191s) [upload=450996]: Done uploading segments... Dec 27 01:26:23.986849720 (513h57m23.857932765s) [upload=450996]: Upload finished. err: <nil> Duration: 30.003055772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX.] duration: 6.105416857s trace: https://jaeger.dev.storj.io/trace/e20afd88f0a89fb FAIL (2025-12-27 01:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:24:51.567489728 (513h55m51.438572763s) [upload=450979]: Starting upload Dec 27 01:24:51.567557076 (513h55m51.438640111s) [upload=450979]: Done starting upload Dec 27 01:24:51.567637088 (513h55m51.438720133s) [upload=450979]: Uploading segments... Dec 27 01:24:51.567780482 (513h55m51.438863527s) [upload=450979]: Got next segment. Inline: false Dec 27 01:24:51.567837059 (513h55m51.438920104s) [upload=450979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:24:51.567846026 (513h55m51.438929091s) [upload=450979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:24:51.815021971 (513h55m51.686105006s) [seg_pos={0 0}, upload=450979]: Begin upload segment... Dec 27 01:24:51.815066205 (513h55m51.68614924s) [seg_pos={0 0}, upload=450979]: Done begin upload segment. Dec 27 01:24:51.815092726 (513h55m51.686175771s) [noise=true , address=145.79.227.14:4838 , piece_id=NRA3RNJYQQP5LPDXZ2XV5C7GYIZ33IX476PI63V6VUSDDM3EM6TA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.815317825 (513h55m51.68640086s) [noise=true , address=45.91.116.9:4264 , piece_id=3EX3DMKIZC4AFQTVYG5X76C565EIDOGY3I6CK6HR446VX5EYXL6A, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.815689913 (513h55m51.686772958s) [noise=true , address=145.79.227.15:4873 , piece_id=TSWPAD2G7OZRORYP7PFTV2QLU5PJ7L252DT7Z3ZZTBNANRDWDI5Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.816364677 (513h55m51.687447712s) [noise=true , address=145.79.227.50:4375 , piece_id=VAYBAQLT4GMZL27BORKKSMRAICDZQNVOYAW6UG7OIFCGRCZLWB7Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.817172927 (513h55m51.688255972s) [noise=true , address=145.79.227.42:4123 , piece_id=3Q3PSXQVUF673VSER4VM3QLWDONBCNSGLDBKTY3LMIRVNKS2LNTA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.817707114 (513h55m51.688790159s) [noise=true , address=77.90.31.17:4773 , piece_id=VFTEQT5AV3YVRE4JBCY6SNYZBRJCJW3GSGC3B6OCBX4BF72Z7BNA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.818293390 (513h55m51.689376435s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.818930163 (513h55m51.690013208s) [noise=true , address=145.79.227.17:4972 , piece_id=LI6ZD25PY3KKYJ7RNDIGHQFRVBEQ7KGM54VWLWWAA66D2BQ7ULQA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.819537338 (513h55m51.690620373s) [noise=true , address=145.79.227.49:4337 , piece_id=GBQJMYM3WHIBO76B4EWZUXZG2ODLHWJTMIVAFRMS7JJNTSE2CQRQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.820096925 (513h55m51.69117997s) [noise=true , address=145.79.227.39:4014 , piece_id=RB7SMJMKEMC4DYQNRGO7DJVGNJZ7YW7ABAN6YIQOZSNIO7UUGDZQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.820563853 (513h55m51.691646899s) [noise=true , address=145.79.227.51:4411 , piece_id=JW25EWCXKIEPJGZDNXXX34ZDSR4UKJZPQ5JHB24NU4KKMJIDWYCQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821045721 (513h55m51.692128756s) [noise=true , address=145.79.227.11:4731 , piece_id=BBLBFRI3VJMOZKRYVS2VWAJPQERSFQKCA5HDACCKZRDSVOENNZKQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821456804 (513h55m51.692539849s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821861354 (513h55m51.692944399s) [noise=true , address=145.79.227.43:4158 , piece_id=W2XSYV3KKUE6UQ2KKFJ7SY7LYXHPEFOKB3FPWIASDVZUCRWBOIYQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.822263670 (513h55m51.693346705s) [noise=true , address=145.79.227.3:4150 , piece_id=LYMBULFOAOKUGLAV7EOMWPOHMFNQC4QXXEVEBIFHZVWGFWYY6N7Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.822716292 (513h55m51.693799327s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:53.529277377 (513h55m53.400360412s) [upload=450979]: Next returned nil segment Dec 27 01:24:53.529324959 (513h55m53.400407994s) [upload=450979]: Waiting for error group managing segments... Dec 27 01:24:53.808242422 (513h55m53.679325467s) [noise=true , address=145.79.227.11:4731 , piece_id=BBLBFRI3VJMOZKRYVS2VWAJPQERSFQKCA5HDACCKZRDSVOENNZKQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.819346565 (513h55m53.690429611s) [noise=true , address=145.79.227.50:4375 , piece_id=VAYBAQLT4GMZL27BORKKSMRAICDZQNVOYAW6UG7OIFCGRCZLWB7Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.823853258 (513h55m53.694936303s) [noise=true , address=145.79.227.49:4337 , piece_id=GBQJMYM3WHIBO76B4EWZUXZG2ODLHWJTMIVAFRMS7JJNTSE2CQRQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.834616121 (513h55m53.705699166s) [noise=true , address=145.79.227.15:4873 , piece_id=TSWPAD2G7OZRORYP7PFTV2QLU5PJ7L252DT7Z3ZZTBNANRDWDI5Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.835948248 (513h55m53.707031283s) [noise=true , address=145.79.227.39:4014 , piece_id=RB7SMJMKEMC4DYQNRGO7DJVGNJZ7YW7ABAN6YIQOZSNIO7UUGDZQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.839452169 (513h55m53.710535214s) [noise=true , address=145.79.227.3:4150 , piece_id=LYMBULFOAOKUGLAV7EOMWPOHMFNQC4QXXEVEBIFHZVWGFWYY6N7Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.843155280 (513h55m53.714238325s) [noise=true , address=145.79.227.17:4972 , piece_id=LI6ZD25PY3KKYJ7RNDIGHQFRVBEQ7KGM54VWLWWAA66D2BQ7ULQA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.845555511 (513h55m53.716638546s) [noise=true , address=145.79.227.14:4838 , piece_id=NRA3RNJYQQP5LPDXZ2XV5C7GYIZ33IX476PI63V6VUSDDM3EM6TA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.846182385 (513h55m53.71726543s) [noise=true , address=145.79.227.51:4411 , piece_id=JW25EWCXKIEPJGZDNXXX34ZDSR4UKJZPQ5JHB24NU4KKMJIDWYCQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.853688358 (513h55m53.724771403s) [noise=true , address=145.79.227.42:4123 , piece_id=3Q3PSXQVUF673VSER4VM3QLWDONBCNSGLDBKTY3LMIRVNKS2LNTA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.856464443 (513h55m53.727547488s) [noise=true , address=145.79.227.43:4158 , piece_id=W2XSYV3KKUE6UQ2KKFJ7SY7LYXHPEFOKB3FPWIASDVZUCRWBOIYQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.998440363 (513h55m53.869523408s) [noise=true , address=77.90.31.17:4773 , piece_id=VFTEQT5AV3YVRE4JBCY6SNYZBRJCJW3GSGC3B6OCBX4BF72Z7BNA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:54.052246246 (513h55m53.923329291s) [noise=true , address=45.91.116.9:4264 , piece_id=3EX3DMKIZC4AFQTVYG5X76C565EIDOGY3I6CK6HR446VX5EYXL6A, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:54.052273188 (513h55m53.923356233s) [seg_pos={0 0}, upload=450979]: Segment reached optimal threshold of 13 pieces. Dec 27 01:24:54.052539455 (513h55m53.92362249s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:54.052548753 (513h55m53.923631798s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052620319 (513h55m53.923703364s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:48754->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 01:24:54.052630247 (513h55m53.923713282s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052901244 (513h55m53.923984279s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:54.052912084 (513h55m53.923995129s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052938825 (513h55m53.92402187s) [seg_pos={0 0}, upload=450979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:24:54.053032444 (513h55m53.924115479s) [upload=450979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:24:54.053039707 (513h55m53.924122742s) [upload=450979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:24:54.345925789 (513h55m54.217008834s) [upload=450979]: Done uploading segments... Dec 27 01:24:54.345947912 (513h55m54.217030957s) [upload=450979]: Upload finished. err: <nil> Duration: 6.105443308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXXX] duration: 30.003370279s trace: https://jaeger.dev.storj.io/trace/e20afd88f0a89fb FAIL (2025-12-27 01:25:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:24:51.567489728 (513h55m51.438572763s) [upload=450979]: Starting upload Dec 27 01:24:51.567557076 (513h55m51.438640111s) [upload=450979]: Done starting upload Dec 27 01:24:51.567637088 (513h55m51.438720133s) [upload=450979]: Uploading segments... Dec 27 01:24:51.567780482 (513h55m51.438863527s) [upload=450979]: Got next segment. Inline: false Dec 27 01:24:51.567837059 (513h55m51.438920104s) [upload=450979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:24:51.567846026 (513h55m51.438929091s) [upload=450979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:24:51.815021971 (513h55m51.686105006s) [seg_pos={0 0}, upload=450979]: Begin upload segment... Dec 27 01:24:51.815066205 (513h55m51.68614924s) [seg_pos={0 0}, upload=450979]: Done begin upload segment. Dec 27 01:24:51.815092726 (513h55m51.686175771s) [noise=true , address=145.79.227.14:4838 , piece_id=NRA3RNJYQQP5LPDXZ2XV5C7GYIZ33IX476PI63V6VUSDDM3EM6TA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.815317825 (513h55m51.68640086s) [noise=true , address=45.91.116.9:4264 , piece_id=3EX3DMKIZC4AFQTVYG5X76C565EIDOGY3I6CK6HR446VX5EYXL6A, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.815689913 (513h55m51.686772958s) [noise=true , address=145.79.227.15:4873 , piece_id=TSWPAD2G7OZRORYP7PFTV2QLU5PJ7L252DT7Z3ZZTBNANRDWDI5Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.816364677 (513h55m51.687447712s) [noise=true , address=145.79.227.50:4375 , piece_id=VAYBAQLT4GMZL27BORKKSMRAICDZQNVOYAW6UG7OIFCGRCZLWB7Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.817172927 (513h55m51.688255972s) [noise=true , address=145.79.227.42:4123 , piece_id=3Q3PSXQVUF673VSER4VM3QLWDONBCNSGLDBKTY3LMIRVNKS2LNTA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.817707114 (513h55m51.688790159s) [noise=true , address=77.90.31.17:4773 , piece_id=VFTEQT5AV3YVRE4JBCY6SNYZBRJCJW3GSGC3B6OCBX4BF72Z7BNA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.818293390 (513h55m51.689376435s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.818930163 (513h55m51.690013208s) [noise=true , address=145.79.227.17:4972 , piece_id=LI6ZD25PY3KKYJ7RNDIGHQFRVBEQ7KGM54VWLWWAA66D2BQ7ULQA, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.819537338 (513h55m51.690620373s) [noise=true , address=145.79.227.49:4337 , piece_id=GBQJMYM3WHIBO76B4EWZUXZG2ODLHWJTMIVAFRMS7JJNTSE2CQRQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.820096925 (513h55m51.69117997s) [noise=true , address=145.79.227.39:4014 , piece_id=RB7SMJMKEMC4DYQNRGO7DJVGNJZ7YW7ABAN6YIQOZSNIO7UUGDZQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.820563853 (513h55m51.691646899s) [noise=true , address=145.79.227.51:4411 , piece_id=JW25EWCXKIEPJGZDNXXX34ZDSR4UKJZPQ5JHB24NU4KKMJIDWYCQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821045721 (513h55m51.692128756s) [noise=true , address=145.79.227.11:4731 , piece_id=BBLBFRI3VJMOZKRYVS2VWAJPQERSFQKCA5HDACCKZRDSVOENNZKQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821456804 (513h55m51.692539849s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.821861354 (513h55m51.692944399s) [noise=true , address=145.79.227.43:4158 , piece_id=W2XSYV3KKUE6UQ2KKFJ7SY7LYXHPEFOKB3FPWIASDVZUCRWBOIYQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.822263670 (513h55m51.693346705s) [noise=true , address=145.79.227.3:4150 , piece_id=LYMBULFOAOKUGLAV7EOMWPOHMFNQC4QXXEVEBIFHZVWGFWYY6N7Q, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:51.822716292 (513h55m51.693799327s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Uploading piece... Dec 27 01:24:53.529277377 (513h55m53.400360412s) [upload=450979]: Next returned nil segment Dec 27 01:24:53.529324959 (513h55m53.400407994s) [upload=450979]: Waiting for error group managing segments... Dec 27 01:24:53.808242422 (513h55m53.679325467s) [noise=true , address=145.79.227.11:4731 , piece_id=BBLBFRI3VJMOZKRYVS2VWAJPQERSFQKCA5HDACCKZRDSVOENNZKQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.819346565 (513h55m53.690429611s) [noise=true , address=145.79.227.50:4375 , piece_id=VAYBAQLT4GMZL27BORKKSMRAICDZQNVOYAW6UG7OIFCGRCZLWB7Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.823853258 (513h55m53.694936303s) [noise=true , address=145.79.227.49:4337 , piece_id=GBQJMYM3WHIBO76B4EWZUXZG2ODLHWJTMIVAFRMS7JJNTSE2CQRQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.834616121 (513h55m53.705699166s) [noise=true , address=145.79.227.15:4873 , piece_id=TSWPAD2G7OZRORYP7PFTV2QLU5PJ7L252DT7Z3ZZTBNANRDWDI5Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.835948248 (513h55m53.707031283s) [noise=true , address=145.79.227.39:4014 , piece_id=RB7SMJMKEMC4DYQNRGO7DJVGNJZ7YW7ABAN6YIQOZSNIO7UUGDZQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.839452169 (513h55m53.710535214s) [noise=true , address=145.79.227.3:4150 , piece_id=LYMBULFOAOKUGLAV7EOMWPOHMFNQC4QXXEVEBIFHZVWGFWYY6N7Q, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.843155280 (513h55m53.714238325s) [noise=true , address=145.79.227.17:4972 , piece_id=LI6ZD25PY3KKYJ7RNDIGHQFRVBEQ7KGM54VWLWWAA66D2BQ7ULQA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.845555511 (513h55m53.716638546s) [noise=true , address=145.79.227.14:4838 , piece_id=NRA3RNJYQQP5LPDXZ2XV5C7GYIZ33IX476PI63V6VUSDDM3EM6TA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.846182385 (513h55m53.71726543s) [noise=true , address=145.79.227.51:4411 , piece_id=JW25EWCXKIEPJGZDNXXX34ZDSR4UKJZPQ5JHB24NU4KKMJIDWYCQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.853688358 (513h55m53.724771403s) [noise=true , address=145.79.227.42:4123 , piece_id=3Q3PSXQVUF673VSER4VM3QLWDONBCNSGLDBKTY3LMIRVNKS2LNTA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.856464443 (513h55m53.727547488s) [noise=true , address=145.79.227.43:4158 , piece_id=W2XSYV3KKUE6UQ2KKFJ7SY7LYXHPEFOKB3FPWIASDVZUCRWBOIYQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:53.998440363 (513h55m53.869523408s) [noise=true , address=77.90.31.17:4773 , piece_id=VFTEQT5AV3YVRE4JBCY6SNYZBRJCJW3GSGC3B6OCBX4BF72Z7BNA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:54.052246246 (513h55m53.923329291s) [noise=true , address=45.91.116.9:4264 , piece_id=3EX3DMKIZC4AFQTVYG5X76C565EIDOGY3I6CK6HR446VX5EYXL6A, seg_pos={0 0}, upload=450979]: Done uploading piece. err: <nil> Dec 27 01:24:54.052273188 (513h55m53.923356233s) [seg_pos={0 0}, upload=450979]: Segment reached optimal threshold of 13 pieces. Dec 27 01:24:54.052539455 (513h55m53.92362249s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:54.052548753 (513h55m53.923631798s) [noise=true , address=145.79.227.34:4905 , piece_id=HMSDTB7Q4R43ZJKZ4XBAXY4M52CQUPKVPFT2CEF7DD5NLDGHRHXQ, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052620319 (513h55m53.923703364s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:48754->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 01:24:54.052630247 (513h55m53.923713282s) [noise=true , address=45.91.116.13:4154 , piece_id=CWFXEU2OG3KS3UVNQU4CVWU527APLJK25G3COA5UPRFUR5R2BLYA, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052901244 (513h55m53.923984279s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:54.052912084 (513h55m53.923995129s) [noise=true , address=45.91.116.12:4227 , piece_id=SGHJCPBCD2PTHOOVI4ZZTB4LIGP4NO74R5PEL54PG6TAHJG7CQAQ, seg_pos={0 0}, upload=450979]: Long tail cancellation due to optimal threshold Dec 27 01:24:54.052938825 (513h55m53.92402187s) [seg_pos={0 0}, upload=450979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:24:54.053032444 (513h55m53.924115479s) [upload=450979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:24:54.053039707 (513h55m53.924122742s) [upload=450979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:24:54.345925789 (513h55m54.217008834s) [upload=450979]: Done uploading segments... Dec 27 01:24:54.345947912 (513h55m54.217030957s) [upload=450979]: Upload finished. err: <nil> Duration: 30.00338139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXXX] duration: 30.003028887s trace: https://jaeger.dev.storj.io/trace/36e48669515af191 FAIL (2025-12-27 01:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: QDGCMYITJMWTUCMD6NECZS2FCZBRJN27375ZCCS276K6OXBBZE2Q) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: VYJX5H4ULV57D4AVYBAGFLUJIEYPWDXYA5ZV2WX33QERTXZDLNTA) context deadline exceeded ======LOGS====== 27 01:24:21.254496471 (513h55m21.125579506s) [upload=450969]: Starting upload Dec 27 01:24:21.254595269 (513h55m21.125678304s) [upload=450969]: Done starting upload Dec 27 01:24:21.254681554 (513h55m21.125764599s) [upload=450969]: Uploading segments... Dec 27 01:24:21.254824085 (513h55m21.12590713s) [upload=450969]: Got next segment. Inline: false Dec 27 01:24:21.254862218 (513h55m21.125945263s) [upload=450969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:24:21.254868029 (513h55m21.125951074s) [upload=450969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:24:21.498982022 (513h55m21.370065067s) [seg_pos={0 0}, upload=450969]: Begin upload segment... Dec 27 01:24:21.499044462 (513h55m21.370127497s) [seg_pos={0 0}, upload=450969]: Done begin upload segment. Dec 27 01:24:21.499095890 (513h55m21.370178925s) [noise=true , address=145.79.227.17:4971 , piece_id=5KSQCN2PLLB25DGFQHW6Z5VI2O4KBPVABWW6YS2SBB4256AM2AEQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.499113202 (513h55m21.370196248s) [noise=true , address=45.91.116.12:4225 , piece_id=LZ5IYHMFU6XUEHVHGX3WR2YPGJIHADGWN3TSEPQPZD7A4RYDPSRQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.499697926 (513h55m21.370780971s) [noise=true , address=145.79.227.34:4907 , piece_id=NXO4SMDKF4FJIOP43WLOZSTBSMBB5ZBKH3ADTRVJTMQ5TSDUPXZQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.500084562 (513h55m21.371167597s) [noise=true , address=45.91.116.14:4192 , piece_id=QDGCMYITJMWTUCMD6NECZS2FCZBRJN27375ZCCS276K6OXBBZE2Q, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.500482861 (513h55m21.371565906s) [noise=true , address=45.91.116.7:4012 , piece_id=XO66SJUQMJKBQA276U2GZQIGA47XG2D2XQTMDY2F5W2VUMYKHREQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.500890777 (513h55m21.371973812s) [noise=true , address=145.79.227.16:4912 , piece_id=7BKTY7L7WQJYFQDWB4U44TGO2HMYCIW7R3W5MUA5LER47SFHJX4Q, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.501554371 (513h55m21.372637416s) [noise=true , address=45.91.116.5:4299 , piece_id=VYJX5H4ULV57D4AVYBAGFLUJIEYPWDXYA5ZV2WX33QERTXZDLNTA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.502027091 (513h55m21.373110126s) [noise=true , address=45.91.116.8:4046 , piece_id=4ERPTUPD4DFUCK2P5TX7MJUZCJYYATEZMU6IDQVQ5T2HVWXA2QUA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.502335068 (513h55m21.373418113s) [noise=true , address=77.90.31.13:4629 , piece_id=3YXUTMJAZHT2CBQJW554XJFR6W66TZH7F6NTBODPEPRPQTYFYOWA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.503583214 (513h55m21.374666249s) [noise=true , address=77.90.31.17:4770 , piece_id=72YWRBUU2LKRRAXFU2GBQDSUDSMTQYL5LVW5GQIJFEZIPVIPDU7A, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.504623304 (513h55m21.375706339s) [noise=true , address=145.79.227.47:4266 , piece_id=I2ICB3FKZRYQ5B3QUKZ7LW6NUWTU2CQVQD7YQ6GTC4KRPPHGWACA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.505620924 (513h55m21.376703959s) [noise=true , address=145.79.227.2:4114 , piece_id=IDVL574SWDWC4LH4BDAUHKGGCOXQNTMM4UDBQYKBMGHM2P3QRR3Q, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.505965621 (513h55m21.377048666s) [noise=true , address=145.79.227.49:4338 , piece_id=23XNFJNBZM2UEJWPR5ZD2YBSDPHKQRFP7EMJOICNKZNRHLNQJAKA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.506718133 (513h55m21.377801168s) [noise=true , address=145.79.227.51:4411 , piece_id=XJOXSNJCIZ6BE4EYQEBBS5AJFF5J55G5NBSZ25KNOIDKV37ZZLSQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.507526733 (513h55m21.378609778s) [noise=true , address=145.79.227.50:4373 , piece_id=ES2G5CRXSP26TGVGQPY7OXM7UX4UM55HOWQYS52MWP7ZLIKGDVCQ, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:21.508058346 (513h55m21.379141381s) [noise=true , address=145.79.227.36:4519 , piece_id=PXMQYZEZ4HEZ5DJUTHM6KMA5UQQJIIM6OUXHCSZITCLG7ZOUYKBA, seg_pos={0 0}, upload=450969]: Uploading piece... Dec 27 01:24:23.139396770 (513h55m23.010479815s) [upload=450969]: Next returned nil segment Dec 27 01:24:23.139423250 (513h55m23.010506295s) [upload=450969]: Waiting for error group managing segments... Dec 27 01:24:23.523917385 (513h55m23.39500042s) [noise=true , address=145.79.227.51:4411 , piece_id=XJOXSNJCIZ6BE4EYQEBBS5AJFF5J55G5NBSZ25KNOIDKV37ZZLSQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.531811687 (513h55m23.402894722s) [noise=true , address=145.79.227.49:4338 , piece_id=23XNFJNBZM2UEJWPR5ZD2YBSDPHKQRFP7EMJOICNKZNRHLNQJAKA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.532168796 (513h55m23.403251831s) [noise=true , address=145.79.227.50:4373 , piece_id=ES2G5CRXSP26TGVGQPY7OXM7UX4UM55HOWQYS52MWP7ZLIKGDVCQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.532672125 (513h55m23.40375517s) [noise=true , address=145.79.227.17:4971 , piece_id=5KSQCN2PLLB25DGFQHW6Z5VI2O4KBPVABWW6YS2SBB4256AM2AEQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.539745564 (513h55m23.410828609s) [noise=true , address=145.79.227.47:4266 , piece_id=I2ICB3FKZRYQ5B3QUKZ7LW6NUWTU2CQVQD7YQ6GTC4KRPPHGWACA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.539760713 (513h55m23.410843758s) [noise=true , address=145.79.227.36:4519 , piece_id=PXMQYZEZ4HEZ5DJUTHM6KMA5UQQJIIM6OUXHCSZITCLG7ZOUYKBA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.575398151 (513h55m23.446481196s) [noise=true , address=145.79.227.16:4912 , piece_id=7BKTY7L7WQJYFQDWB4U44TGO2HMYCIW7R3W5MUA5LER47SFHJX4Q, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.709093783 (513h55m23.580176818s) [noise=true , address=77.90.31.17:4770 , piece_id=72YWRBUU2LKRRAXFU2GBQDSUDSMTQYL5LVW5GQIJFEZIPVIPDU7A, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.846589392 (513h55m23.717672427s) [noise=true , address=45.91.116.14:4192 , piece_id=QDGCMYITJMWTUCMD6NECZS2FCZBRJN27375ZCCS276K6OXBBZE2Q, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.851043794 (513h55m23.722126839s) [noise=true , address=45.91.116.7:4012 , piece_id=XO66SJUQMJKBQA276U2GZQIGA47XG2D2XQTMDY2F5W2VUMYKHREQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.874925889 (513h55m23.746008924s) [noise=true , address=45.91.116.12:4225 , piece_id=LZ5IYHMFU6XUEHVHGX3WR2YPGJIHADGWN3TSEPQPZD7A4RYDPSRQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.901071402 (513h55m23.772154437s) [noise=true , address=145.79.227.34:4907 , piece_id=NXO4SMDKF4FJIOP43WLOZSTBSMBB5ZBKH3ADTRVJTMQ5TSDUPXZQ, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.989355455 (513h55m23.8604385s) [noise=true , address=45.91.116.5:4299 , piece_id=VYJX5H4ULV57D4AVYBAGFLUJIEYPWDXYA5ZV2WX33QERTXZDLNTA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: <nil> Dec 27 01:24:23.989377237 (513h55m23.860460272s) [seg_pos={0 0}, upload=450969]: Segment reached optimal threshold of 13 pieces. Dec 27 01:24:23.989568791 (513h55m23.860651826s) [noise=true , address=45.91.116.8:4046 , piece_id=4ERPTUPD4DFUCK2P5TX7MJUZCJYYATEZMU6IDQVQ5T2HVWXA2QUA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:23.989575475 (513h55m23.86065851s) [noise=true , address=45.91.116.8:4046 , piece_id=4ERPTUPD4DFUCK2P5TX7MJUZCJYYATEZMU6IDQVQ5T2HVWXA2QUA, seg_pos={0 0}, upload=450969]: Long tail cancellation due to optimal threshold Dec 27 01:24:23.989842042 (513h55m23.860925087s) [noise=true , address=145.79.227.2:4114 , piece_id=IDVL574SWDWC4LH4BDAUHKGGCOXQNTMM4UDBQYKBMGHM2P3QRR3Q, seg_pos={0 0}, upload=450969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:42416->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:42416->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42416->145.79.227.2:4114: use of closed network connection Dec 27 01:24:23.989852011 (513h55m23.860935047s) [noise=true , address=145.79.227.2:4114 , piece_id=IDVL574SWDWC4LH4BDAUHKGGCOXQNTMM4UDBQYKBMGHM2P3QRR3Q, seg_pos={0 0}, upload=450969]: Long tail cancellation due to optimal threshold Dec 27 01:24:23.989917255 (513h55m23.86100029s) [noise=true , address=77.90.31.13:4629 , piece_id=3YXUTMJAZHT2CBQJW554XJFR6W66TZH7F6NTBODPEPRPQTYFYOWA, seg_pos={0 0}, upload=450969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 01:24:23.989924139 (513h55m23.861007184s) [noise=true , address=77.90.31.13:4629 , piece_id=3YXUTMJAZHT2CBQJW554XJFR6W66TZH7F6NTBODPEPRPQTYFYOWA, seg_pos={0 0}, upload=450969]: Long tail cancellation due to optimal threshold Dec 27 01:24:23.989941251 (513h55m23.861024286s) [seg_pos={0 0}, upload=450969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:24:23.990017907 (513h55m23.861100952s) [upload=450969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:24:23.990024239 (513h55m23.861107275s) [upload=450969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:24:24.286069346 (513h55m24.157152401s) [upload=450969]: Done uploading segments... Dec 27 01:24:24.286095256 (513h55m24.157178301s) [upload=450969]: Upload finished. err: <nil> Duration: 30.003054927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXXX] duration: 30.003442972s trace: https://jaeger.dev.storj.io/trace/5f5246cad4cbd693 FAIL (2025-12-27 01:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :23:22.834929862 (513h54m22.706012897s) [upload=450955]: Starting upload Dec 27 01:23:22.835159610 (513h54m22.706242645s) [upload=450955]: Done starting upload Dec 27 01:23:22.835225395 (513h54m22.70630844s) [upload=450955]: Uploading segments... Dec 27 01:23:22.835405758 (513h54m22.706488804s) [upload=450955]: Got next segment. Inline: false Dec 27 01:23:22.835468077 (513h54m22.706551122s) [upload=450955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:23:22.835474930 (513h54m22.706557985s) [upload=450955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:23:23.082031946 (513h54m22.953114991s) [seg_pos={0 0}, upload=450955]: Begin upload segment... Dec 27 01:23:23.082080549 (513h54m22.953163594s) [seg_pos={0 0}, upload=450955]: Done begin upload segment. Dec 27 01:23:23.082110566 (513h54m22.953193611s) [noise=true , address=145.79.227.14:4837 , piece_id=PN553FAXROWPCUUSY52BVEP7XND4CSTTTL7RQDOW4WF2IGG7JJXA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.082495599 (513h54m22.953578644s) [noise=true , address=145.79.227.38:4979 , piece_id=DA6I7HN5HNN6SCGU5WO6JD5DLSOVEIB3OM7G3PPKSRP6IPYRCK4A, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.083476096 (513h54m22.954559131s) [noise=true , address=77.90.31.13:4627 , piece_id=UWSZI6HJKOEGHJCL4MYPKO5BJRRXHS57QGT7BJ2WPMMXUWPSQHOA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.083990976 (513h54m22.955074021s) [noise=true , address=45.91.116.12:4226 , piece_id=RMUU2LEDH7YYNA6HSSHDIODJIZQ4UN4ZUK25VWUCGUKHAAE42MTQ, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.084438147 (513h54m22.955521182s) [noise=true , address=145.79.227.51:4411 , piece_id=JZMWSSMQPPWHRY6IZOXG3UAGOLXMA4NS4AZQMFE4JKTYWEKUZZ7A, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.084925946 (513h54m22.956008981s) [noise=true , address=145.79.227.12:4767 , piece_id=32L6G45DAAQ7YI3FZMOLMYDTTLD4R2C55MZI5EGG7I3ENA2WN37A, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.085266094 (513h54m22.956349139s) [noise=true , address=145.79.227.41:4086 , piece_id=ONPVJB3S7QUYWOTKMACQA6QYBVAU7J5EUGQSOGOXC64G5DL3ULXQ, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.086919602 (513h54m22.958002647s) [noise=true , address=45.91.116.14:4190 , piece_id=AB66P5OBY5REUK2P5GCJXZ2FI6HOVLBOIEILB67IMBUN545WQSIA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.087349652 (513h54m22.958432697s) [noise=true , address=145.79.227.45:4230 , piece_id=7ICHLMNVAWJR7IZZW6BS2TSDVNHT3V3T74CHFR6NCBV4T6DXJR2A, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.090864554 (513h54m22.961947599s) [noise=true , address=145.79.227.15:4875 , piece_id=DBUTHZ6KTKYEBWCSWBXTVL3KJS3KSICBVS2QIDA2UQRXGO7RNLOA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.094225253 (513h54m22.965308299s) [noise=true , address=77.90.31.10:4517 , piece_id=HUFZ7VPGWOGNSR2LDS46MAZJY2JZ5AIMGPWUBEIS6ORPVDIKB3IQ, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.096400054 (513h54m22.967483089s) [noise=true , address=77.90.31.16:4736 , piece_id=O67VWAC6CLQV6FWTTZH2OPESBDANO7NS6OYIL6XOTGT7DZ44CZCA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.097840848 (513h54m22.968923893s) [noise=true , address=77.90.31.14:4664 , piece_id=TIRXE4F5UYGTOBJJYI5FIXFBJGYPVJFER2YYQCYVQ2VTW7FUCDVA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.099211638 (513h54m22.970294683s) [noise=true , address=77.90.31.11:4555 , piece_id=UCMMVQ4OJ5AIKQ47SSUPEDOESRAZ6WHXHOFCF7EFNJIXQNSLXF4A, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.103587700 (513h54m22.974670745s) [noise=true , address=145.79.227.3:4150 , piece_id=UECMRRKWYUV7U5DXNW3BM5OEFJN56DGHZ2Z23GNP27JJ73IJSGJA, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:23.105398689 (513h54m22.976481754s) [noise=true , address=45.91.116.4:4937 , piece_id=32OHPVSYCGAUGGCGBC6ZNMAYFG6RO3FQIYXPZCVSALALIPFMMLCQ, seg_pos={0 0}, upload=450955]: Uploading piece... Dec 27 01:23:24.714604368 (513h54m24.585687403s) [upload=450955]: Next returned nil segment Dec 27 01:23:24.714631099 (513h54m24.585714144s) [upload=450955]: Waiting for error group managing segments... Dec 27 01:23:25.097774993 (513h54m24.968858038s) [noise=true , address=145.79.227.14:4837 , piece_id=PN553FAXROWPCUUSY52BVEP7XND4CSTTTL7RQDOW4WF2IGG7JJXA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.098199922 (513h54m24.969282957s) [noise=true , address=145.79.227.51:4411 , piece_id=JZMWSSMQPPWHRY6IZOXG3UAGOLXMA4NS4AZQMFE4JKTYWEKUZZ7A, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.102011950 (513h54m24.973094985s) [noise=true , address=145.79.227.38:4979 , piece_id=DA6I7HN5HNN6SCGU5WO6JD5DLSOVEIB3OM7G3PPKSRP6IPYRCK4A, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.107083888 (513h54m24.978166933s) [noise=true , address=145.79.227.45:4230 , piece_id=7ICHLMNVAWJR7IZZW6BS2TSDVNHT3V3T74CHFR6NCBV4T6DXJR2A, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.113705426 (513h54m24.984788461s) [noise=true , address=145.79.227.12:4767 , piece_id=32L6G45DAAQ7YI3FZMOLMYDTTLD4R2C55MZI5EGG7I3ENA2WN37A, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.135012528 (513h54m25.006095573s) [noise=true , address=145.79.227.15:4875 , piece_id=DBUTHZ6KTKYEBWCSWBXTVL3KJS3KSICBVS2QIDA2UQRXGO7RNLOA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.285780829 (513h54m25.156863875s) [noise=true , address=77.90.31.16:4736 , piece_id=O67VWAC6CLQV6FWTTZH2OPESBDANO7NS6OYIL6XOTGT7DZ44CZCA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.325560265 (513h54m25.1966433s) [noise=true , address=145.79.227.41:4086 , piece_id=ONPVJB3S7QUYWOTKMACQA6QYBVAU7J5EUGQSOGOXC64G5DL3ULXQ, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.333491977 (513h54m25.204575022s) [noise=true , address=45.91.116.12:4226 , piece_id=RMUU2LEDH7YYNA6HSSHDIODJIZQ4UN4ZUK25VWUCGUKHAAE42MTQ, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.366145133 (513h54m25.237228178s) [noise=true , address=77.90.31.11:4555 , piece_id=UCMMVQ4OJ5AIKQ47SSUPEDOESRAZ6WHXHOFCF7EFNJIXQNSLXF4A, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.396062259 (513h54m25.267145294s) [noise=true , address=77.90.31.13:4627 , piece_id=UWSZI6HJKOEGHJCL4MYPKO5BJRRXHS57QGT7BJ2WPMMXUWPSQHOA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.414738390 (513h54m25.285821425s) [noise=true , address=45.91.116.14:4190 , piece_id=AB66P5OBY5REUK2P5GCJXZ2FI6HOVLBOIEILB67IMBUN545WQSIA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.594605640 (513h54m25.465688685s) [noise=true , address=77.90.31.10:4517 , piece_id=HUFZ7VPGWOGNSR2LDS46MAZJY2JZ5AIMGPWUBEIS6ORPVDIKB3IQ, seg_pos={0 0}, upload=450955]: Done uploading piece. err: <nil> Dec 27 01:23:25.594642991 (513h54m25.465726036s) [seg_pos={0 0}, upload=450955]: Segment reached optimal threshold of 13 pieces. Dec 27 01:23:25.594925440 (513h54m25.466008515s) [noise=true , address=145.79.227.3:4150 , piece_id=UECMRRKWYUV7U5DXNW3BM5OEFJN56DGHZ2Z23GNP27JJ73IJSGJA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:51584->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:51584->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51584->145.79.227.3:4150: use of closed network connection Dec 27 01:23:25.594939336 (513h54m25.466022411s) [noise=true , address=145.79.227.3:4150 , piece_id=UECMRRKWYUV7U5DXNW3BM5OEFJN56DGHZ2Z23GNP27JJ73IJSGJA, seg_pos={0 0}, upload=450955]: Long tail cancellation due to optimal threshold Dec 27 01:23:25.595014178 (513h54m25.466097213s) [noise=true , address=45.91.116.4:4937 , piece_id=32OHPVSYCGAUGGCGBC6ZNMAYFG6RO3FQIYXPZCVSALALIPFMMLCQ, seg_pos={0 0}, upload=450955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 01:23:25.595049836 (513h54m25.466132881s) [noise=true , address=45.91.116.4:4937 , piece_id=32OHPVSYCGAUGGCGBC6ZNMAYFG6RO3FQIYXPZCVSALALIPFMMLCQ, seg_pos={0 0}, upload=450955]: Long tail cancellation due to optimal threshold Dec 27 01:23:25.595363623 (513h54m25.466446658s) [noise=true , address=77.90.31.14:4664 , piece_id=TIRXE4F5UYGTOBJJYI5FIXFBJGYPVJFER2YYQCYVQ2VTW7FUCDVA, seg_pos={0 0}, upload=450955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 27 01:23:25.595374093 (513h54m25.466457128s) [noise=true , address=77.90.31.14:4664 , piece_id=TIRXE4F5UYGTOBJJYI5FIXFBJGYPVJFER2YYQCYVQ2VTW7FUCDVA, seg_pos={0 0}, upload=450955]: Long tail cancellation due to optimal threshold Dec 27 01:23:25.595426022 (513h54m25.466509067s) [seg_pos={0 0}, upload=450955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:23:25.595599843 (513h54m25.466682888s) [upload=450955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:23:25.595608880 (513h54m25.466691925s) [upload=450955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:23:25.892678308 (513h54m25.763761353s) [upload=450955]: Done uploading segments... Dec 27 01:23:25.892699258 (513h54m25.763782293s) [upload=450955]: Upload finished. err: <nil> Duration: 30.003469933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXXX] duration: 30.003762408s trace: https://jaeger.dev.storj.io/trace/7aa3f07d47010e32 FAIL (2025-12-27 01:23:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VG6KXOC4TZ3U6FZBSQFK6GCZKJMOLQYYUBUI6S5HHTWWF42S4OZQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: CDT5YDHMLILS4JEL4IROCAUOB3ZUPAN2AFT6QRISNUADILPWNG4Q) context deadline exceeded ======LOGS====== gments... Dec 27 01:22:33.558566324 (513h53m33.429649359s) [upload=450946]: Got next segment. Inline: false Dec 27 01:22:33.558593236 (513h53m33.429676271s) [upload=450946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:22:33.558597143 (513h53m33.429680178s) [upload=450946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:22:33.803539404 (513h53m33.674622439s) [seg_pos={0 0}, upload=450946]: Begin upload segment... Dec 27 01:22:33.803606732 (513h53m33.674689777s) [seg_pos={0 0}, upload=450946]: Done begin upload segment. Dec 27 01:22:33.803633784 (513h53m33.674716819s) [noise=true , address=145.79.227.42:4123 , piece_id=J5T3QXLLYXEFMMSPPTMOFANS4KSO4LFQ6U45ZRQYBZE6A6SLOZEQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.803648161 (513h53m33.674731206s) [noise=true , address=45.91.116.7:4012 , piece_id=4V37SPPPUA64VEO2KKF2SKNRH63KTYCMDGO6TRQEO6GUMUU5YQ6Q, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.804225831 (513h53m33.675308866s) [noise=true , address=145.79.227.17:4972 , piece_id=KJFF5ZRJP2LGKYG35FH2ZWYHZF755KL2C7RBMQNJX5MXDL7Z67GQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.804755870 (513h53m33.675838915s) [noise=true , address=145.79.227.2:4114 , piece_id=VWLISANQFU7MQXOBL2QMSSMYT6FFYAK7CPWQVCI4WOYR2F3P4ALA, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.805252406 (513h53m33.676335451s) [noise=true , address=45.91.116.8:4046 , piece_id=Y55KO5T5PYJDMMP6COTAAX3ZQC5GYDQTH5MCR6RFFW4DOPX6LQAQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.805727150 (513h53m33.676810185s) [noise=true , address=145.79.227.51:4410 , piece_id=VJ7JMDELJZ4GIMHTLW2KRZVJ7UFHNBGYMXDOXE6V5TRJDK2TDS7A, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.806227663 (513h53m33.677310698s) [noise=true , address=45.91.116.9:4263 , piece_id=KLGEMPDIIO7MWCH4QJ42OXQHOEHZHF4VX63JI6TQCPU3THB7GPVA, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.806764655 (513h53m33.67784769s) [noise=true , address=45.91.116.5:4297 , piece_id=NUD5DRVSQNJJAEF4GGEX4JEHCC4A7JUWCIBX7UCQJA3GDK7I65FQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.808114716 (513h53m33.679197751s) [noise=true , address=145.79.227.15:4875 , piece_id=R43ELKHFLRXIQF2XCXFJOB2SPQUDENDQAWG6IE7MVAPW75V6NOTA, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.809503821 (513h53m33.680586856s) [noise=true , address=145.79.227.48:4303 , piece_id=WLBRR7JGM7GQXRPOYM5UKN64EZQEXTPEDPP2YMRXHAMTQEDFZE3Q, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.810869722 (513h53m33.681952757s) [noise=true , address=77.90.31.10:4519 , piece_id=VG6KXOC4TZ3U6FZBSQFK6GCZKJMOLQYYUBUI6S5HHTWWF42S4OZQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.812415605 (513h53m33.68349864s) [noise=true , address=145.79.227.37:4565 , piece_id=BIZGVQ5YGIEUBTW3IFLG5ZK7O65PPX2PZDO6BCC5VYHOGPRKMM4Q, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.813903598 (513h53m33.684986644s) [noise=true , address=145.79.227.11:4731 , piece_id=SEIU4XD63R7QMQSCCW26CYOO56R42YKZQWVZIMHFOWHXIKO3MQIA, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.815671375 (513h53m33.68675443s) [noise=true , address=145.79.227.52:4447 , piece_id=PY6VP2IROQAXU5M4YKTJKNV4RH4BZ3I6IE27G6JTSB6VWGHW3HBQ, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.817048897 (513h53m33.688131932s) [noise=true , address=45.91.116.11:4118 , piece_id=CDT5YDHMLILS4JEL4IROCAUOB3ZUPAN2AFT6QRISNUADILPWNG4Q, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:33.818051617 (513h53m33.689134662s) [noise=true , address=45.91.116.13:4156 , piece_id=METFIUKVJBSFRJ2EANH4ZZFSNS6IP5CPRJGHCSGCK633GOG4BVNA, seg_pos={0 0}, upload=450946]: Uploading piece... Dec 27 01:22:35.526041432 (513h53m35.397124488s) [upload=450946]: Next returned nil segment Dec 27 01:22:35.526081509 (513h53m35.397164544s) [upload=450946]: Waiting for error group managing segments... Dec 27 01:22:35.812428450 (513h53m35.683511485s) [noise=true , address=145.79.227.42:4123 , piece_id=J5T3QXLLYXEFMMSPPTMOFANS4KSO4LFQ6U45ZRQYBZE6A6SLOZEQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.819726546 (513h53m35.690809581s) [noise=true , address=145.79.227.17:4972 , piece_id=KJFF5ZRJP2LGKYG35FH2ZWYHZF755KL2C7RBMQNJX5MXDL7Z67GQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.827245153 (513h53m35.698328188s) [noise=true , address=145.79.227.51:4410 , piece_id=VJ7JMDELJZ4GIMHTLW2KRZVJ7UFHNBGYMXDOXE6V5TRJDK2TDS7A, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.828097605 (513h53m35.69918064s) [noise=true , address=145.79.227.15:4875 , piece_id=R43ELKHFLRXIQF2XCXFJOB2SPQUDENDQAWG6IE7MVAPW75V6NOTA, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.835063029 (513h53m35.706146064s) [noise=true , address=145.79.227.11:4731 , piece_id=SEIU4XD63R7QMQSCCW26CYOO56R42YKZQWVZIMHFOWHXIKO3MQIA, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.836361311 (513h53m35.707444346s) [noise=true , address=145.79.227.48:4303 , piece_id=WLBRR7JGM7GQXRPOYM5UKN64EZQEXTPEDPP2YMRXHAMTQEDFZE3Q, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.859254953 (513h53m35.730337998s) [noise=true , address=145.79.227.52:4447 , piece_id=PY6VP2IROQAXU5M4YKTJKNV4RH4BZ3I6IE27G6JTSB6VWGHW3HBQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:35.859336198 (513h53m35.730419233s) [noise=true , address=145.79.227.37:4565 , piece_id=BIZGVQ5YGIEUBTW3IFLG5ZK7O65PPX2PZDO6BCC5VYHOGPRKMM4Q, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.013123788 (513h53m35.884206823s) [noise=true , address=77.90.31.10:4519 , piece_id=VG6KXOC4TZ3U6FZBSQFK6GCZKJMOLQYYUBUI6S5HHTWWF42S4OZQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.030485978 (513h53m35.901569013s) [noise=true , address=45.91.116.9:4263 , piece_id=KLGEMPDIIO7MWCH4QJ42OXQHOEHZHF4VX63JI6TQCPU3THB7GPVA, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.063543764 (513h53m35.934626799s) [noise=true , address=45.91.116.11:4118 , piece_id=CDT5YDHMLILS4JEL4IROCAUOB3ZUPAN2AFT6QRISNUADILPWNG4Q, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.173509653 (513h53m36.044592698s) [noise=true , address=45.91.116.13:4156 , piece_id=METFIUKVJBSFRJ2EANH4ZZFSNS6IP5CPRJGHCSGCK633GOG4BVNA, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.176554501 (513h53m36.047637536s) [noise=true , address=45.91.116.5:4297 , piece_id=NUD5DRVSQNJJAEF4GGEX4JEHCC4A7JUWCIBX7UCQJA3GDK7I65FQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: <nil> Dec 27 01:22:36.176569359 (513h53m36.047652394s) [seg_pos={0 0}, upload=450946]: Segment reached optimal threshold of 13 pieces. Dec 27 01:22:36.176742770 (513h53m36.047825805s) [noise=true , address=45.91.116.8:4046 , piece_id=Y55KO5T5PYJDMMP6COTAAX3ZQC5GYDQTH5MCR6RFFW4DOPX6LQAQ, seg_pos={0 0}, upload=450946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46902->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46902->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46902->45.91.116.8:4046: use of closed network connection Dec 27 01:22:36.176748911 (513h53m36.047831946s) [noise=true , address=45.91.116.8:4046 , piece_id=Y55KO5T5PYJDMMP6COTAAX3ZQC5GYDQTH5MCR6RFFW4DOPX6LQAQ, seg_pos={0 0}, upload=450946]: Long tail cancellation due to optimal threshold Dec 27 01:22:36.176853120 (513h53m36.047936155s) [noise=true , address=145.79.227.2:4114 , piece_id=VWLISANQFU7MQXOBL2QMSSMYT6FFYAK7CPWQVCI4WOYR2F3P4ALA, seg_pos={0 0}, upload=450946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:60330->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:60330->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60330->145.79.227.2:4114: use of closed network connection Dec 27 01:22:36.176868789 (513h53m36.047951824s) [noise=true , address=145.79.227.2:4114 , piece_id=VWLISANQFU7MQXOBL2QMSSMYT6FFYAK7CPWQVCI4WOYR2F3P4ALA, seg_pos={0 0}, upload=450946]: Long tail cancellation due to optimal threshold Dec 27 01:22:36.177068279 (513h53m36.048151314s) [noise=true , address=45.91.116.7:4012 , piece_id=4V37SPPPUA64VEO2KKF2SKNRH63KTYCMDGO6TRQEO6GUMUU5YQ6Q, seg_pos={0 0}, upload=450946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 01:22:36.177075202 (513h53m36.048158237s) [noise=true , address=45.91.116.7:4012 , piece_id=4V37SPPPUA64VEO2KKF2SKNRH63KTYCMDGO6TRQEO6GUMUU5YQ6Q, seg_pos={0 0}, upload=450946]: Long tail cancellation due to optimal threshold Dec 27 01:22:36.177091213 (513h53m36.048174258s) [seg_pos={0 0}, upload=450946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:22:36.177168821 (513h53m36.048251856s) [upload=450946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:22:36.177174982 (513h53m36.048258017s) [upload=450946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:22:36.469452837 (513h53m36.340535872s) [upload=450946]: Done uploading segments... Dec 27 01:22:36.469471441 (513h53m36.340554476s) [upload=450946]: Upload finished. err: <nil> Duration: 30.003787135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.XX.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXXX] duration: 30.004995835s trace: https://jaeger.dev.storj.io/trace/6599e6d6b3d506b FAIL (2025-12-27 01:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: JWJ3WJOFH5OIR5U6743IHUKOVPBFL3OREWBR5ZTAJ5QRA6TN6MDA) context deadline exceeded; piecestore: (Node ID: 1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, Piece ID: ULWRKBY6E6MPT3PEKMMH2WIWI45HBMOBHYT2FZKCM4MXXENQQX4Q) context deadline exceeded; piecestore: (Node ID: 12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, Piece ID: YZQ5Y337NYQJU7ADGRVDTTAVLZSF5EXJPOKCW6SX3PXRIOGMJRSA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: F7W7P267AIDNUYPF6USGVFXCR3C5NERBI4ITUNR2GS6DTCRNIOAA) context deadline exceeded ======LOGS====== Dec 27 01:21:42.911634096 (513h52m42.782717131s) [upload=450934]: Starting upload Dec 27 01:21:42.911701865 (513h52m42.7827849s) [upload=450934]: Done starting upload Dec 27 01:21:42.911902968 (513h52m42.782986003s) [upload=450934]: Uploading segments... Dec 27 01:21:42.912000454 (513h52m42.783083499s) [upload=450934]: Got next segment. Inline: false Dec 27 01:21:42.912090164 (513h52m42.783173209s) [upload=450934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:21:42.912096497 (513h52m42.783179532s) [upload=450934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:21:43.156689422 (513h52m43.027772457s) [seg_pos={0 0}, upload=450934]: Begin upload segment... Dec 27 01:21:43.156740800 (513h52m43.027823835s) [seg_pos={0 0}, upload=450934]: Done begin upload segment. Dec 27 01:21:43.156827545 (513h52m43.02791058s) [noise=true , address=145.79.227.47:4267 , piece_id=6LM2ESPHLBF2SG5M67G35ZRVCLRHISXRY3LK3FBJHOR7QCQ5XMRQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.156833175 (513h52m43.027916211s) [noise=true , address=77.90.31.11:4556 , piece_id=JWJ3WJOFH5OIR5U6743IHUKOVPBFL3OREWBR5ZTAJ5QRA6TN6MDA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.157346393 (513h52m43.028429428s) [noise=true , address=145.79.227.48:4302 , piece_id=CACERP45SNYR624ED3UVF65OYEP2MZ7CWXW5PLMGHIQOWOWMG3NA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.157682343 (513h52m43.028765408s) [noise=true , address=145.79.227.44:4195 , piece_id=35N6PKWSV53TCZ4UZJ3G6KN5KRKUN6EI6ES4F4U3MIX7NXPPSHXQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.158088185 (513h52m43.02917122s) [noise=true , address=145.79.227.36:4518 , piece_id=SV3INFLYEF725GPU5Y7IB5IPWKPSV5AN2WU72WUKU6AC3BNE4WQA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.158465153 (513h52m43.029548188s) [noise=true , address=145.79.227.15:4873 , piece_id=B73MUJ3ZK2RL7WPZBN7ZHJ4COO6N4LVLTOE6BV4G5NWX3L6OVYTQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.158801173 (513h52m43.029884208s) [noise=true , address=145.79.227.39:4015 , piece_id=7D4AZOMAGHK2YVSGUI7NRX7Z35IGFUSZJEMKUXHV6VVO73HLQPFQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.159126152 (513h52m43.030209187s) [noise=true , address=145.79.227.37:4566 , piece_id=ULWRKBY6E6MPT3PEKMMH2WIWI45HBMOBHYT2FZKCM4MXXENQQX4Q, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.159495385 (513h52m43.03057842s) [noise=true , address=145.79.227.19:4077 , piece_id=DFJSB7LFK4JUJ2UTE5B4DRN43LR7FR6Q5OQSIEV5FXSNTBQ7TWRQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.160523703 (513h52m43.031606748s) [noise=true , address=45.91.116.8:4048 , piece_id=VJNKKK6XO4PW7AA3YB7TPHML5TSHCKFH5TZ5LYW75TAAP56IN4WA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.161165265 (513h52m43.0322483s) [noise=true , address=45.91.116.4:4940 , piece_id=YZQ5Y337NYQJU7ADGRVDTTAVLZSF5EXJPOKCW6SX3PXRIOGMJRSA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.161558604 (513h52m43.032641649s) [noise=true , address=45.91.116.11:4118 , piece_id=F7W7P267AIDNUYPF6USGVFXCR3C5NERBI4ITUNR2GS6DTCRNIOAA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.162011306 (513h52m43.033094341s) [noise=true , address=145.79.227.11:4730 , piece_id=CFRBTVA2F3SB6BXBMWO3LQCINIAVWOCUE6BC2O3PRLLUH27XWIDA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.164398090 (513h52m43.035481125s) [noise=true , address=145.79.227.5:4222 , piece_id=JBERIDNA22QBQK72BHVAQ5K3WEMIYMD7O5BPHN77TS2RBE6VRNUA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.172994349 (513h52m43.044077384s) [noise=true , address=145.79.227.14:4838 , piece_id=OXDALBCQWX3H2N2FCX67H5JBDJPGPYTISTUDFVHQOI3CRQA4QMCA, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:43.174942198 (513h52m43.046025233s) [noise=true , address=145.79.227.38:4977 , piece_id=QGSM2N7KRGOKLUM3CMIH4R54ZSMJRNTM3XGIYPDQNJIS27QNJGLQ, seg_pos={0 0}, upload=450934]: Uploading piece... Dec 27 01:21:44.870774544 (513h52m44.741857589s) [upload=450934]: Next returned nil segment Dec 27 01:21:44.870790515 (513h52m44.74187355s) [upload=450934]: Waiting for error group managing segments... Dec 27 01:21:45.159819712 (513h52m45.030902778s) [noise=true , address=145.79.227.11:4730 , piece_id=CFRBTVA2F3SB6BXBMWO3LQCINIAVWOCUE6BC2O3PRLLUH27XWIDA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.170598937 (513h52m45.041681972s) [noise=true , address=145.79.227.15:4873 , piece_id=B73MUJ3ZK2RL7WPZBN7ZHJ4COO6N4LVLTOE6BV4G5NWX3L6OVYTQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.175177104 (513h52m45.04626015s) [noise=true , address=145.79.227.38:4977 , piece_id=QGSM2N7KRGOKLUM3CMIH4R54ZSMJRNTM3XGIYPDQNJIS27QNJGLQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.178049183 (513h52m45.049132218s) [noise=true , address=145.79.227.47:4267 , piece_id=6LM2ESPHLBF2SG5M67G35ZRVCLRHISXRY3LK3FBJHOR7QCQ5XMRQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.183084511 (513h52m45.054167546s) [noise=true , address=145.79.227.37:4566 , piece_id=ULWRKBY6E6MPT3PEKMMH2WIWI45HBMOBHYT2FZKCM4MXXENQQX4Q, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.189598725 (513h52m45.06068176s) [noise=true , address=145.79.227.39:4015 , piece_id=7D4AZOMAGHK2YVSGUI7NRX7Z35IGFUSZJEMKUXHV6VVO73HLQPFQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.197360364 (513h52m45.068443399s) [noise=true , address=145.79.227.14:4838 , piece_id=OXDALBCQWX3H2N2FCX67H5JBDJPGPYTISTUDFVHQOI3CRQA4QMCA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.205966561 (513h52m45.077049596s) [noise=true , address=145.79.227.36:4518 , piece_id=SV3INFLYEF725GPU5Y7IB5IPWKPSV5AN2WU72WUKU6AC3BNE4WQA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.373537909 (513h52m45.244620944s) [noise=true , address=45.91.116.4:4940 , piece_id=YZQ5Y337NYQJU7ADGRVDTTAVLZSF5EXJPOKCW6SX3PXRIOGMJRSA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.400767548 (513h52m45.271850593s) [noise=true , address=77.90.31.11:4556 , piece_id=JWJ3WJOFH5OIR5U6743IHUKOVPBFL3OREWBR5ZTAJ5QRA6TN6MDA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.488983201 (513h52m45.360066236s) [noise=true , address=45.91.116.8:4048 , piece_id=VJNKKK6XO4PW7AA3YB7TPHML5TSHCKFH5TZ5LYW75TAAP56IN4WA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:45.787794486 (513h52m45.658877521s) [noise=true , address=145.79.227.48:4302 , piece_id=CACERP45SNYR624ED3UVF65OYEP2MZ7CWXW5PLMGHIQOWOWMG3NA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:46.407385756 (513h52m46.278468791s) [noise=true , address=45.91.116.11:4118 , piece_id=F7W7P267AIDNUYPF6USGVFXCR3C5NERBI4ITUNR2GS6DTCRNIOAA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: <nil> Dec 27 01:21:46.407406816 (513h52m46.278489851s) [seg_pos={0 0}, upload=450934]: Segment reached optimal threshold of 13 pieces. Dec 27 01:21:46.407641954 (513h52m46.278724989s) [noise=true , address=145.79.227.5:4222 , piece_id=JBERIDNA22QBQK72BHVAQ5K3WEMIYMD7O5BPHN77TS2RBE6VRNUA, seg_pos={0 0}, upload=450934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 27 01:21:46.407649408 (513h52m46.278732443s) [noise=true , address=145.79.227.5:4222 , piece_id=JBERIDNA22QBQK72BHVAQ5K3WEMIYMD7O5BPHN77TS2RBE6VRNUA, seg_pos={0 0}, upload=450934]: Long tail cancellation due to optimal threshold Dec 27 01:21:46.407768715 (513h52m46.27885175s) [noise=true , address=145.79.227.44:4195 , piece_id=35N6PKWSV53TCZ4UZJ3G6KN5KRKUN6EI6ES4F4U3MIX7NXPPSHXQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 01:21:46.407789274 (513h52m46.278872309s) [noise=true , address=145.79.227.44:4195 , piece_id=35N6PKWSV53TCZ4UZJ3G6KN5KRKUN6EI6ES4F4U3MIX7NXPPSHXQ, seg_pos={0 0}, upload=450934]: Long tail cancellation due to optimal threshold Dec 27 01:21:46.407948287 (513h52m46.279031332s) [noise=true , address=145.79.227.19:4077 , piece_id=DFJSB7LFK4JUJ2UTE5B4DRN43LR7FR6Q5OQSIEV5FXSNTBQ7TWRQ, seg_pos={0 0}, upload=450934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 27 01:21:46.407956943 (513h52m46.279039988s) [noise=true , address=145.79.227.19:4077 , piece_id=DFJSB7LFK4JUJ2UTE5B4DRN43LR7FR6Q5OQSIEV5FXSNTBQ7TWRQ, seg_pos={0 0}, upload=450934]: Long tail cancellation due to optimal threshold Dec 27 01:21:46.407978114 (513h52m46.279061159s) [seg_pos={0 0}, upload=450934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:21:46.408063877 (513h52m46.279146922s) [upload=450934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:21:46.408068936 (513h52m46.279151971s) [upload=450934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:21:46.691581420 (513h52m46.562664465s) [upload=450934]: Done uploading segments... Dec 27 01:21:46.691603060 (513h52m46.562686095s) [upload=450934]: Upload finished. err: <nil> Duration: 30.005022096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [..XX.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXXX] duration: 30.00320197s trace: https://jaeger.dev.storj.io/trace/61e3376f953386aa FAIL (2025-12-27 01:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6BMCASW626M3ETTYEMP5WFWBPWNJVT353N42IZCIWNJH34WBEJQQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: MQCWRGYXLRF5PXJPMUMTCHNDY7I4FUMW5OZ7QLHSRUYYQBUGNPYQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: IRQ5ME4UIO4QG4ALH7MMS6L36NEPDIBQNEFL2AC7XBAVWI2Y6FUQ) context deadline exceeded ======LOGS====== ts... Dec 27 01:20:49.687959558 (513h51m49.559042593s) [upload=450923]: Got next segment. Inline: false Dec 27 01:20:49.688004603 (513h51m49.559087638s) [upload=450923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:20:49.688011596 (513h51m49.559094641s) [upload=450923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:20:49.932615393 (513h51m49.803698438s) [seg_pos={0 0}, upload=450923]: Begin upload segment... Dec 27 01:20:49.932672612 (513h51m49.803755647s) [seg_pos={0 0}, upload=450923]: Done begin upload segment. Dec 27 01:20:49.932699263 (513h51m49.803782298s) [noise=true , address=45.91.116.6:4974 , piece_id=IRQ5ME4UIO4QG4ALH7MMS6L36NEPDIBQNEFL2AC7XBAVWI2Y6FUQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.932782631 (513h51m49.803865676s) [noise=true , address=145.79.227.47:4267 , piece_id=YYRPZK26XREIHNQYO5CV33MTWE7EXHI6FHOQQPKAHJK2JWQJWVNA, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.933303503 (513h51m49.804386538s) [noise=true , address=45.91.116.14:4192 , piece_id=C7J66QQ6RFVG6VQMEKZN3W6OH4LPS3356UN6MA3RDRK7YSFLGLBQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.933590210 (513h51m49.804673255s) [noise=true , address=145.79.227.19:4077 , piece_id=BGQUCD7TJDQNLX6BW6AEWU3QO5BCJMCKO5TYT5M5FDWAVHMVUAEA, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.933952719 (513h51m49.805035755s) [noise=true , address=77.90.31.21:4878 , piece_id=E36EQHIRCBZ5MNS27WKIDCN7FPKYXR2ARJKCL4UKBWSIP5BBKOUQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.934275273 (513h51m49.805358308s) [noise=true , address=45.91.116.8:4046 , piece_id=H255AMKOETZSQ33ZEFYDKQ75SPIHAYRQEALHXR3DWWUJD7JMUQ3A, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.934575946 (513h51m49.805658991s) [noise=true , address=145.79.227.34:4906 , piece_id=X74BBHQDISM7KLHFCWLMAZRV43OV3Y5H44OOCXYIYR44RVDJ75BQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.935116034 (513h51m49.806199069s) [noise=true , address=145.79.227.2:4114 , piece_id=GT2RJZ57PDCB6EMYIQB4NFHEFVK5Q5MBH7LU7P2WHA223CVPXLNQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.935407410 (513h51m49.806490445s) [noise=true , address=145.79.227.29:4941 , piece_id=FT63CHE4P4CYFEGHJ4SZB6ZXSA64FDESJTGUHY3EXJMMBHOHG63Q, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.935630855 (513h51m49.80671389s) [noise=true , address=145.79.227.51:4410 , piece_id=4YHL4356M63JCXGHL2ZVXURG664JFVFMYXH4BWPV46CIZBRPZBPA, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.935698063 (513h51m49.806781098s) [noise=true , address=45.91.116.5:4299 , piece_id=AUB565T5W3KVHIVVHR325SSBLAHVC5VGAG6P3ZZMIQBMFJ4OGEYQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.935989898 (513h51m49.807072933s) [noise=true , address=145.79.227.16:4912 , piece_id=LCQYHWMSWZVVGLHBIUZLIK4Q7HW5IILSTQ7KVZXQC27TM4YUPVCQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.936118884 (513h51m49.807201919s) [noise=true , address=145.79.227.5:4222 , piece_id=RV7IQIGQRERQQBCORQ52FWNH7TZEWQCZS5VWKZYKZ4GIXWDSNEMA, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.936369601 (513h51m49.807452636s) [noise=true , address=145.79.227.36:4517 , piece_id=LHVYBPUPWQF3ELK2TRJABHGVNV54MPT72HSYT5YYDIBMHS72GCBA, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.936492776 (513h51m49.807575811s) [noise=true , address=45.91.116.9:4263 , piece_id=6BMCASW626M3ETTYEMP5WFWBPWNJVT353N42IZCIWNJH34WBEJQQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:49.936844626 (513h51m49.807927671s) [noise=true , address=45.91.116.12:4226 , piece_id=MQCWRGYXLRF5PXJPMUMTCHNDY7I4FUMW5OZ7QLHSRUYYQBUGNPYQ, seg_pos={0 0}, upload=450923]: Uploading piece... Dec 27 01:20:51.649214693 (513h51m51.520297728s) [upload=450923]: Next returned nil segment Dec 27 01:20:51.649256492 (513h51m51.520339537s) [upload=450923]: Waiting for error group managing segments... Dec 27 01:20:51.941010538 (513h51m51.812093583s) [noise=true , address=145.79.227.36:4517 , piece_id=LHVYBPUPWQF3ELK2TRJABHGVNV54MPT72HSYT5YYDIBMHS72GCBA, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:51.941466917 (513h51m51.812549962s) [noise=true , address=145.79.227.16:4912 , piece_id=LCQYHWMSWZVVGLHBIUZLIK4Q7HW5IILSTQ7KVZXQC27TM4YUPVCQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:51.955756396 (513h51m51.826839441s) [noise=true , address=145.79.227.51:4410 , piece_id=4YHL4356M63JCXGHL2ZVXURG664JFVFMYXH4BWPV46CIZBRPZBPA, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:51.962150310 (513h51m51.833233346s) [noise=true , address=145.79.227.19:4077 , piece_id=BGQUCD7TJDQNLX6BW6AEWU3QO5BCJMCKO5TYT5M5FDWAVHMVUAEA, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:51.966559106 (513h51m51.837642141s) [noise=true , address=145.79.227.29:4941 , piece_id=FT63CHE4P4CYFEGHJ4SZB6ZXSA64FDESJTGUHY3EXJMMBHOHG63Q, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:51.997754415 (513h51m51.86883746s) [noise=true , address=145.79.227.47:4267 , piece_id=YYRPZK26XREIHNQYO5CV33MTWE7EXHI6FHOQQPKAHJK2JWQJWVNA, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.117477163 (513h51m51.988560208s) [noise=true , address=77.90.31.21:4878 , piece_id=E36EQHIRCBZ5MNS27WKIDCN7FPKYXR2ARJKCL4UKBWSIP5BBKOUQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.157016541 (513h51m52.028099586s) [noise=true , address=45.91.116.5:4299 , piece_id=AUB565T5W3KVHIVVHR325SSBLAHVC5VGAG6P3ZZMIQBMFJ4OGEYQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.171136417 (513h51m52.042219462s) [noise=true , address=45.91.116.8:4046 , piece_id=H255AMKOETZSQ33ZEFYDKQ75SPIHAYRQEALHXR3DWWUJD7JMUQ3A, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.205493096 (513h51m52.076576141s) [noise=true , address=45.91.116.6:4974 , piece_id=IRQ5ME4UIO4QG4ALH7MMS6L36NEPDIBQNEFL2AC7XBAVWI2Y6FUQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.209972996 (513h51m52.081056031s) [noise=true , address=45.91.116.9:4263 , piece_id=6BMCASW626M3ETTYEMP5WFWBPWNJVT353N42IZCIWNJH34WBEJQQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.219789257 (513h51m52.090872302s) [noise=true , address=45.91.116.12:4226 , piece_id=MQCWRGYXLRF5PXJPMUMTCHNDY7I4FUMW5OZ7QLHSRUYYQBUGNPYQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.491933174 (513h51m52.363016219s) [noise=true , address=45.91.116.14:4192 , piece_id=C7J66QQ6RFVG6VQMEKZN3W6OH4LPS3356UN6MA3RDRK7YSFLGLBQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: <nil> Dec 27 01:20:52.491960525 (513h51m52.36304357s) [seg_pos={0 0}, upload=450923]: Segment reached optimal threshold of 13 pieces. Dec 27 01:20:52.492206032 (513h51m52.363289067s) [noise=true , address=145.79.227.2:4114 , piece_id=GT2RJZ57PDCB6EMYIQB4NFHEFVK5Q5MBH7LU7P2WHA223CVPXLNQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:48444->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:48444->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48444->145.79.227.2:4114: use of closed network connection Dec 27 01:20:52.492218306 (513h51m52.363301341s) [noise=true , address=145.79.227.2:4114 , piece_id=GT2RJZ57PDCB6EMYIQB4NFHEFVK5Q5MBH7LU7P2WHA223CVPXLNQ, seg_pos={0 0}, upload=450923]: Long tail cancellation due to optimal threshold Dec 27 01:20:52.492341591 (513h51m52.363424636s) [noise=true , address=145.79.227.5:4222 , piece_id=RV7IQIGQRERQQBCORQ52FWNH7TZEWQCZS5VWKZYKZ4GIXWDSNEMA, seg_pos={0 0}, upload=450923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:40696->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:40696->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40696->145.79.227.5:4222: use of closed network connection Dec 27 01:20:52.492361509 (513h51m52.363444554s) [noise=true , address=145.79.227.5:4222 , piece_id=RV7IQIGQRERQQBCORQ52FWNH7TZEWQCZS5VWKZYKZ4GIXWDSNEMA, seg_pos={0 0}, upload=450923]: Long tail cancellation due to optimal threshold Dec 27 01:20:52.492582820 (513h51m52.363665865s) [noise=true , address=145.79.227.34:4906 , piece_id=X74BBHQDISM7KLHFCWLMAZRV43OV3Y5H44OOCXYIYR44RVDJ75BQ, seg_pos={0 0}, upload=450923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 01:20:52.492591858 (513h51m52.363674923s) [noise=true , address=145.79.227.34:4906 , piece_id=X74BBHQDISM7KLHFCWLMAZRV43OV3Y5H44OOCXYIYR44RVDJ75BQ, seg_pos={0 0}, upload=450923]: Long tail cancellation due to optimal threshold Dec 27 01:20:52.492617977 (513h51m52.363701022s) [seg_pos={0 0}, upload=450923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:20:52.492725391 (513h51m52.363808436s) [upload=450923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:20:52.492733427 (513h51m52.363816473s) [upload=450923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:20:52.815381449 (513h51m52.686464514s) [upload=450923]: Done uploading segments... Dec 27 01:20:52.815407980 (513h51m52.686491025s) [upload=450923]: Upload finished. err: <nil> Duration: 30.003224362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [...XX.X..XXX....X...X.X.XXXXX.X..X..X..XXXXX.XXXXXX..XXX.XXX] duration: 30.002869586s trace: https://jaeger.dev.storj.io/trace/438f099b3edd3f88 FAIL (2025-12-27 01:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 01:20:06.307797753 (513h51m6.178880808s) [upload=450911]: Starting upload Dec 27 01:20:06.307904305 (513h51m6.17898735s) [upload=450911]: Done starting upload Dec 27 01:20:06.307988957 (513h51m6.179072002s) [upload=450911]: Uploading segments... Dec 27 01:20:06.308363840 (513h51m6.179446885s) [upload=450911]: Got next segment. Inline: false Dec 27 01:20:06.308417623 (513h51m6.179500658s) [upload=450911]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 01:20:06.308427812 (513h51m6.179510887s) [upload=450911]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 01:20:06.554460639 (513h51m6.425543685s) [seg_pos={0 0}, upload=450911]: Begin upload segment... Dec 27 01:20:06.554538799 (513h51m6.425621844s) [seg_pos={0 0}, upload=450911]: Done begin upload segment. Dec 27 01:20:06.554583644 (513h51m6.425666689s) [noise=true , address=45.91.116.4:4939 , piece_id=57ZXJKGHSKQVJTKOWUMRMGAO6KGZK26RX4367KUNFQBG2RTA4MVQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.554613100 (513h51m6.425696135s) [noise=true , address=145.79.227.29:4942 , piece_id=OD62VUKUHH5G44PWLCA3HPD6WP6CJX6PUZWPT6ADQHQ3OERBGHDQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.554694465 (513h51m6.42577751s) [noise=true , address=145.79.227.35:4482 , piece_id=4COV2LPDGQ3QIHJBWAVACUQBUJMDV6NFJU4K2IBGC463LWMZ2R6Q, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.555156224 (513h51m6.426239269s) [noise=true , address=45.91.116.9:4264 , piece_id=DQYE2B2WNEASAL7REWV7H7PA5JDKJEJMBWQUQBMIX3JYSYFXRZHQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.555593748 (513h51m6.426676793s) [noise=true , address=45.91.116.14:4190 , piece_id=NRUWVIDQLJHHOIOFNZQZ4LJACZJAGXQLFHMBJL6RE6DDX72SPSHQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.555999890 (513h51m6.427082935s) [noise=true , address=45.91.116.5:4300 , piece_id=XHI7L5HALD63OJNZGB4XD5N4I4EJ2QMKQ5E6P5W6IQGP5ZT65DWA, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.556423758 (513h51m6.427506803s) [noise=true , address=145.79.227.15:4874 , piece_id=PLQXGYJWS5ZVXLBA5UMKXI3BSQMQZTT7DHHO6LULCFAUIHZWPLKQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.556917498 (513h51m6.428000533s) [noise=true , address=45.91.116.13:4154 , piece_id=EFRULGVLBHXBZBEL6RQWQL6ZIN54CZOHM7TBN4AZIC3FSIWZZMVQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.557445323 (513h51m6.428528368s) [noise=true , address=45.91.116.12:4227 , piece_id=URYPNUFHGKOGFZHRFNJ6KZYPGCFT7O3COZPZYPC42ZKIIHNVTYHA, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.558176996 (513h51m6.429260051s) [noise=true , address=145.79.227.50:4375 , piece_id=XDKZDWEQKCC4ITNVDHBWD2ICUU4SF3TN7JJA7BXXFEBDPDIVVZEQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.560424285 (513h51m6.43150733s) [noise=true , address=145.79.227.19:4079 , piece_id=EL2J56ZQZ5F3TAD356ZHYLQDBHDA6FMO5FNOOP5APCPUAS6RLDCA, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.560893349 (513h51m6.431976394s) [noise=true , address=145.79.227.36:4518 , piece_id=UDEFPYJQXH3J5Z2EAFDQT5W2LTGOPSCSFCNUCJ3QJLON2MUOSDAQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.563135748 (513h51m6.434218793s) [noise=true , address=77.90.31.21:4880 , piece_id=N47BW64KHBDV7V6ELNWKVEAWBERFTP2NF3RNJP6PBOHEO7THYI5A, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.564955603 (513h51m6.436038648s) [noise=true , address=77.90.31.19:4841 , piece_id=J6RQBVKROEYYREPJAMDF3HU3J7URYHUD6UX3L72JOAG35OVLCYMQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.566735552 (513h51m6.437818597s) [noise=true , address=45.91.116.7:4010 , piece_id=LZNSK2SDSSI44RZC5YXKUVF3AYQRXSJKOKWO5L6KFPXI62J3JHIA, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:06.568833848 (513h51m6.439916913s) [noise=true , address=145.79.227.37:4566 , piece_id=CJ63VZG6UZYOPQHUYRTNAZQPPXX2TYFVKGHJSUND5E3XBQBQK6MQ, seg_pos={0 0}, upload=450911]: Uploading piece... Dec 27 01:20:08.264543450 (513h51m8.135626485s) [upload=450911]: Next returned nil segment Dec 27 01:20:08.264567906 (513h51m8.135650951s) [upload=450911]: Waiting for error group managing segments... Dec 27 01:20:08.559545230 (513h51m8.430628265s) [noise=true , address=145.79.227.50:4375 , piece_id=XDKZDWEQKCC4ITNVDHBWD2ICUU4SF3TN7JJA7BXXFEBDPDIVVZEQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.565743903 (513h51m8.436826939s) [noise=true , address=145.79.227.15:4874 , piece_id=PLQXGYJWS5ZVXLBA5UMKXI3BSQMQZTT7DHHO6LULCFAUIHZWPLKQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.570607794 (513h51m8.44169084s) [noise=true , address=145.79.227.36:4518 , piece_id=UDEFPYJQXH3J5Z2EAFDQT5W2LTGOPSCSFCNUCJ3QJLON2MUOSDAQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.581034219 (513h51m8.452117254s) [noise=true , address=145.79.227.37:4566 , piece_id=CJ63VZG6UZYOPQHUYRTNAZQPPXX2TYFVKGHJSUND5E3XBQBQK6MQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.648378570 (513h51m8.519461605s) [noise=true , address=145.79.227.29:4942 , piece_id=OD62VUKUHH5G44PWLCA3HPD6WP6CJX6PUZWPT6ADQHQ3OERBGHDQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.796649677 (513h51m8.667732722s) [noise=true , address=45.91.116.7:4010 , piece_id=LZNSK2SDSSI44RZC5YXKUVF3AYQRXSJKOKWO5L6KFPXI62J3JHIA, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.803319346 (513h51m8.674402391s) [noise=true , address=45.91.116.13:4154 , piece_id=EFRULGVLBHXBZBEL6RQWQL6ZIN54CZOHM7TBN4AZIC3FSIWZZMVQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.842349575 (513h51m8.713432611s) [noise=true , address=45.91.116.12:4227 , piece_id=URYPNUFHGKOGFZHRFNJ6KZYPGCFT7O3COZPZYPC42ZKIIHNVTYHA, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.847027503 (513h51m8.718110538s) [noise=true , address=45.91.116.5:4300 , piece_id=XHI7L5HALD63OJNZGB4XD5N4I4EJ2QMKQ5E6P5W6IQGP5ZT65DWA, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.864360357 (513h51m8.735443402s) [noise=true , address=145.79.227.35:4482 , piece_id=4COV2LPDGQ3QIHJBWAVACUQBUJMDV6NFJU4K2IBGC463LWMZ2R6Q, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:08.977873850 (513h51m8.848956895s) [noise=true , address=77.90.31.19:4841 , piece_id=J6RQBVKROEYYREPJAMDF3HU3J7URYHUD6UX3L72JOAG35OVLCYMQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:09.058134016 (513h51m8.929217061s) [noise=true , address=77.90.31.21:4880 , piece_id=N47BW64KHBDV7V6ELNWKVEAWBERFTP2NF3RNJP6PBOHEO7THYI5A, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:09.596056218 (513h51m9.467139253s) [noise=true , address=45.91.116.9:4264 , piece_id=DQYE2B2WNEASAL7REWV7H7PA5JDKJEJMBWQUQBMIX3JYSYFXRZHQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: <nil> Dec 27 01:20:09.596095863 (513h51m9.467178898s) [seg_pos={0 0}, upload=450911]: Segment reached optimal threshold of 13 pieces. Dec 27 01:20:09.596298138 (513h51m9.467381183s) [noise=true , address=45.91.116.4:4939 , piece_id=57ZXJKGHSKQVJTKOWUMRMGAO6KGZK26RX4367KUNFQBG2RTA4MVQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:47958->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:47958->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47958->45.91.116.4:4939: use of closed network connection Dec 27 01:20:09.596308488 (513h51m9.467391533s) [noise=true , address=45.91.116.4:4939 , piece_id=57ZXJKGHSKQVJTKOWUMRMGAO6KGZK26RX4367KUNFQBG2RTA4MVQ, seg_pos={0 0}, upload=450911]: Long tail cancellation due to optimal threshold Dec 27 01:20:09.596531211 (513h51m9.467614256s) [noise=true , address=145.79.227.19:4079 , piece_id=EL2J56ZQZ5F3TAD356ZHYLQDBHDA6FMO5FNOOP5APCPUAS6RLDCA, seg_pos={0 0}, upload=450911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 27 01:20:09.596543566 (513h51m9.467626601s) [noise=true , address=145.79.227.19:4079 , piece_id=EL2J56ZQZ5F3TAD356ZHYLQDBHDA6FMO5FNOOP5APCPUAS6RLDCA, seg_pos={0 0}, upload=450911]: Long tail cancellation due to optimal threshold Dec 27 01:20:09.596572630 (513h51m9.467655675s) [noise=true , address=45.91.116.14:4190 , piece_id=NRUWVIDQLJHHOIOFNZQZ4LJACZJAGXQLFHMBJL6RE6DDX72SPSHQ, seg_pos={0 0}, upload=450911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 01:20:09.596582389 (513h51m9.467665434s) [noise=true , address=45.91.116.14:4190 , piece_id=NRUWVIDQLJHHOIOFNZQZ4LJACZJAGXQLFHMBJL6RE6DDX72SPSHQ, seg_pos={0 0}, upload=450911]: Long tail cancellation due to optimal threshold Dec 27 01:20:09.596604170 (513h51m9.467687215s) [seg_pos={0 0}, upload=450911]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 01:20:09.596696356 (513h51m9.467779391s) [upload=450911]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 01:20:09.596716435 (513h51m9.4677995s) [upload=450911]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 01:20:09.897407718 (513h51m9.768490763s) [upload=450911]: Done uploading segments... Dec 27 01:20:09.897434780 (513h51m9.768517815s) [upload=450911]: Upload finished. err: <nil> Duration: 30.002884925s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)