Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.079947341s FAIL (2025-10-25 15:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 6.079980693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.952177702s FAIL (2025-10-25 15:55:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.952206617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.867278436s FAIL (2025-10-25 15:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.867326648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.773331292s FAIL (2025-10-25 15:53:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.773384564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.176181458s FAIL (2025-10-25 15:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 6.176213508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.185975943s FAIL (2025-10-25 15:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 6.18601172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.189425307s FAIL (2025-10-25 15:50:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 6.189450104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.023165226s FAIL (2025-10-25 15:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 6.023200393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.86030559s FAIL (2025-10-25 15:49:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.860335526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.776710329s FAIL (2025-10-25 15:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.77674152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.885187086s FAIL (2025-10-25 15:48:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.885221563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.849728883s FAIL (2025-10-25 15:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.849758148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.923467946s FAIL (2025-10-25 15:46:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 5.923487042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001915053s FAIL (2025-10-25 15:46:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A) context deadline exceeded ======LOGS====== 770h15m8.490609609s) [upload=677465]: Got next segment. Inline: false Oct 25 15:46:23.028545130 (770h15m8.490637603s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 15:46:23.028549678 (770h15m8.490642161s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 15:46:23.313104540 (770h15m8.775197013s) [seg_pos={0 0}, upload=677465]: Begin upload segment... Oct 25 15:46:23.313176907 (770h15m8.77526938s) [seg_pos={0 0}, upload=677465]: Done begin upload segment. Oct 25 15:46:23.313217484 (770h15m8.775310068s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.313933338 (770h15m8.776025811s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314076721 (770h15m8.776169204s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314435584 (770h15m8.776528057s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314487393 (770h15m8.776579866s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314713583 (770h15m8.776806056s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.314913334 (770h15m8.777005807s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315066134 (770h15m8.777158608s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315258571 (770h15m8.777351084s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315488268 (770h15m8.777580781s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.315740309 (770h15m8.777832782s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316142484 (770h15m8.778234967s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.316535964 (770h15m8.778628447s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317137820 (770h15m8.779230293s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317293466 (770h15m8.779385949s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:23.317683007 (770h15m8.77977548s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Uploading piece... Oct 25 15:46:25.049345446 (770h15m10.511437929s) [upload=677465]: Next returned nil segment Oct 25 15:46:25.049377918 (770h15m10.511470401s) [upload=677465]: Waiting for error group managing segments... Oct 25 15:46:25.325760294 (770h15m10.787852778s) [noise=true , address=145.79.227.10:4402 , piece_id=3X7RRFCV4FRBQ2W5XBPYJYTXAPRSOTB3IZVBYEX2L7PZBWNZCNAA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.344223872 (770h15m10.806316365s) [noise=true , address=145.79.227.7:4295 , piece_id=OX3CIL6IZ6ZVQJKYDYMPTJYVTLFZLR77NT6QCTC2RTEF6ZL7IKMA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.345093528 (770h15m10.807186011s) [noise=true , address=145.79.227.2:4115 , piece_id=MCURXIMLSQRNCIQPWTOBQEKBNKXCEJP2PUAOSA5Z7R2LYSNER6YQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.351771585 (770h15m10.813864058s) [noise=true , address=145.79.227.23:4547 , piece_id=GKXSU2XDKNFADRCMH4I3PXOPQISTLMPUJFEDOH4UQFAV2YZNDISQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.374174775 (770h15m10.836267248s) [noise=true , address=145.79.227.11:4732 , piece_id=A5ILYM33QJBA2BA3LDQ3OZJ6DFAIZDSKPSZDM5USE32N3PICE7RA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.388139316 (770h15m10.850231789s) [noise=true , address=145.79.227.3:4149 , piece_id=KORYLRCDVSUDWDXIYCLYVM3KDCA6YSJBPM6Z44PO7NVFDM57O5TA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.420298894 (770h15m10.882391378s) [noise=true , address=145.79.227.19:4078 , piece_id=CV5MOL7TVGE3N3IFT5OFAKC6JQ5ENOXQY3EO6V2ZERA7D5LWUJEQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.422360790 (770h15m10.884453263s) [noise=true , address=145.79.227.5:4223 , piece_id=VALQBEICIFDVFFJSJQPPF4ICMR43KNPDH36JFW2PXB4IHEQK53SA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.426679014 (770h15m10.888771487s) [noise=true , address=145.79.227.13:4804 , piece_id=XKBEFSPQ7UOXYAPRRYQ76FAR7VY4WRKM4BIMJAZUOBRDJTUFGUCQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.437342328 (770h15m10.899434812s) [noise=true , address=145.79.227.20:4438 , piece_id=3BHBEZFDTYTLRZZ4FU5VZLL24W55JTI6TVBEUFME57ZF33CHCZEA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.623389970 (770h15m11.085482443s) [noise=true , address=77.90.31.19:4841 , piece_id=JCNTTOYTVPCGJCKVIWQ7WSF5SBILB2PDM5ZII3F3N2NOZHXTOI5A, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.641353515 (770h15m11.103445998s) [noise=true , address=77.90.31.17:4772 , piece_id=NYWZOK5Q6AZ3F45A7CAO4BZC3X5K725HZVGZXMIOPJNPGNA4ZFQA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715951253 (770h15m11.178043726s) [noise=true , address=77.90.31.12:4589 , piece_id=DFPQY43H7RVE3KYQO3XPBYDTJLMH6TDATZ2FXDBTCG6HQAHK7TKQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: <nil> Oct 25 15:46:25.715978956 (770h15m11.178071439s) [seg_pos={0 0}, upload=677465]: Segment reached optimal threshold of 13 pieces. Oct 25 15:46:25.716168096 (770h15m11.178260589s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48524->77.90.31.11:4553: use of closed network connection Oct 25 15:46:25.716179509 (770h15m11.178271992s) [noise=true , address=77.90.31.11:4553 , piece_id=NPF6KID3QVGWB5YVU4SKBUIIVXXP4HGTFPD2DHGY3JIOUUIXPVPA, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716228491 (770h15m11.178320974s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.8:4045: use of closed network connection Oct 25 15:46:25.716239081 (770h15m11.178331564s) [noise=true , address=45.91.116.8:4045 , piece_id=TZ4Q6LTOJXJRSBOUAT3IPNRHBAOHKUYEM5NUIPHAVHYHPOTDJB7Q, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716349041 (770h15m11.178441525s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW): protocol: write tcp 188.166.225.170:33862->45.91.116.4:4938: use of closed network connection; context canceled Oct 25 15:46:25.716356455 (770h15m11.178448938s) [noise=true , address=45.91.116.4:4938 , piece_id=IFXYOHXIXKNDRTJHRYMTGAADHUYFIZRT5MBK36AGA5ULTMBCBXJQ, seg_pos={0 0}, upload=677465]: Long tail cancellation due to optimal threshold Oct 25 15:46:25.716368648 (770h15m11.178461121s) [seg_pos={0 0}, upload=677465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 15:46:25.716430256 (770h15m11.178522729s) [upload=677465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 15:46:25.716435415 (770h15m11.178527889s) [upload=677465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 15:46:26.010542744 (770h15m11.472635227s) [upload=677465]: Done uploading segments... Oct 25 15:46:26.010565246 (770h15m11.472657729s) [upload=677465]: Upload finished. err: <nil> Duration: 30.001941242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.916351922s FAIL (2025-10-25 15:12:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.916433398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.955046948s FAIL (2025-10-25 15:12:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.955070392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.974525735s FAIL (2025-10-25 15:11:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.974556354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.078721066s FAIL (2025-10-25 15:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.078755891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.972236101s FAIL (2025-10-25 15:09:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.972258954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.984676877s FAIL (2025-10-25 15:08:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.984710592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.896535509s FAIL (2025-10-25 15:07:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.896584753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.847628123s FAIL (2025-10-25 15:07:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.847659863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.11737246s FAIL (2025-10-25 15:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.11740937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.111361364s FAIL (2025-10-25 15:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.111397794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.097762551s FAIL (2025-10-25 15:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.097792969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.202251468s FAIL (2025-10-25 15:03:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.202289662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.896062024s FAIL (2025-10-25 15:02:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.896093915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.947751101s FAIL (2025-10-25 15:02:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.947783572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.910456574s FAIL (2025-10-25 15:01:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.910543511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.030951911s FAIL (2025-10-25 15:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.030985103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.889374432s FAIL (2025-10-25 14:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.889420831s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.798809607s FAIL (2025-10-25 14:59:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.798846578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.913000883s FAIL (2025-10-25 14:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.913038844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.849164284s FAIL (2025-10-25 14:57:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.849202907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.955565152s FAIL (2025-10-25 14:56:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.955600849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.126846211s FAIL (2025-10-25 14:55:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.126886238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 9.823101396s FAIL (2025-10-25 14:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 9.823133867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 15.000720521s FAIL (2025-10-25 14:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 15.000752851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 19.743463063s FAIL (2025-10-25 14:53:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 19.743499723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.921814533s FAIL (2025-10-25 14:52:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.921839339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.902793231s FAIL (2025-10-25 14:52:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.90282931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 21.729751259s FAIL (2025-10-25 14:51:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 21.729784393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.019271031s FAIL (2025-10-25 14:50:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.019425717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.35589374s FAIL (2025-10-25 14:49:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.355923326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.160048414s FAIL (2025-10-25 14:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.160080807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.507565538s FAIL (2025-10-25 14:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.507643246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.363845641s FAIL (2025-10-25 14:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.363878693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.864177145s FAIL (2025-10-25 14:47:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.864347129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 14.919151146s FAIL (2025-10-25 14:46:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 14.919180692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.7998373s FAIL (2025-10-25 14:45:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.799870863s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)