Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.998796503s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:19:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.998824577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.867026389s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:18:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.867054523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.114957065s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:17:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.114991128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.828092324s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:17:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.828121229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.731110462s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.73113606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.880245043s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:16:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.880275009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.867010348s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:15:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.867039494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.271299559s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 8.271327661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.786772543s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.786803534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.923193395s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:12:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.923219834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.927043809s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:11:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.927074877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 12.240935259s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:11:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 12.24097192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.123989442s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:10:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.124016493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 8.91044642s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:09:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 8.910468952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.932572784s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.932603222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.941124067s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.941153073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.105045464s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.105072364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.769190339s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:07:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.76921178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.776786842s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.77680702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 27.084416637s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:07:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 27.084508722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.978268046s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.978297461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.812353417s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:05:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.812385307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.784073988s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.784103715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.825712575s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:03:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.825742511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.456392123s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.456423082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.816295584s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 12:01:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.816323587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.74557513s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:59:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.745610376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.781153118s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:59:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.781179939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.753259382s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.753289499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.821046549s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.821071988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.69351737s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.693546668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.654889039s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:56:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.654923625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.789921242s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:55:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.789953585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................X................] duration: 6.533391662s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:54:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.533420298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................................X...............] duration: 5.797575887s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:53:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.797604961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................................X..............] duration: 5.827414677s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:52:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.827461186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................................X.............] duration: 5.847263141s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:51:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.847291084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................................X............] duration: 5.957128402s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:50:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.957146165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................................X...........] duration: 5.822643269s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.822675049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................................X..........] duration: 5.981052248s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:49:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.981078578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 5.772017454s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.772047562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................................X........] duration: 5.840362689s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.840393928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................................X.......] duration: 5.764488706s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:47:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.764519393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 5.970163785s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:46:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.970194725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 5.747129241s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.747158347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................................X....] duration: 5.996508632s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:44:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.996538168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........................................X...] duration: 6.076433334s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:43:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 6.076461376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........................................X..] duration: 5.823982798s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:42:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.824007525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................................X.] duration: 5.848469728s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 5.848484324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 30.002823534s trace: https://jaeger.dev.storj.io/trace/52e2e885f3ca3849 FAIL (2025-12-18 11:41:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ) context deadline exceeded ======LOGS====== Dec 18 11:40:41.681580475 (308h11m41.55266352s) [upload=270271]: Starting upload Dec 18 11:40:41.681678993 (308h11m41.552762038s) [upload=270271]: Done starting upload Dec 18 11:40:41.681845320 (308h11m41.552928385s) [upload=270271]: Uploading segments... Dec 18 11:40:41.681992019 (308h11m41.553075064s) [upload=270271]: Got next segment. Inline: false Dec 18 11:40:41.682050521 (308h11m41.553133576s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 11:40:41.682057454 (308h11m41.553140499s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 11:40:41.927719313 (308h11m41.798802359s) [seg_pos={0 0}, upload=270271]: Begin upload segment... Dec 18 11:40:41.927780731 (308h11m41.798863776s) [seg_pos={0 0}, upload=270271]: Done begin upload segment. Dec 18 11:40:41.927821248 (308h11m41.798904293s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929069324 (308h11m41.800152369s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.929571340 (308h11m41.800654385s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930014454 (308h11m41.801097499s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.930470192 (308h11m41.801553237s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931042893 (308h11m41.802125938s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931493129 (308h11m41.802576174s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.931937756 (308h11m41.803020801s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.933547661 (308h11m41.804630706s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.935665594 (308h11m41.806748629s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.937769962 (308h11m41.808853007s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940299578 (308h11m41.811382623s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.940552450 (308h11m41.811635495s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.942212370 (308h11m41.813295415s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.943457061 (308h11m41.814540096s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:41.944579348 (308h11m41.815662393s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Uploading piece... Dec 18 11:40:43.546947811 (308h11m43.418030856s) [upload=270271]: Next returned nil segment Dec 18 11:40:43.546976435 (308h11m43.418059471s) [upload=270271]: Waiting for error group managing segments... Dec 18 11:40:43.926266230 (308h11m43.797349265s) [noise=true , address=145.79.227.14:4839 , piece_id=YE3KFJNHVSZ7YTYSTLV5EIM35EDCIHSZOAG2AXNMPSXQZHKIXTFQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.931481140 (308h11m43.802564185s) [noise=true , address=145.79.227.42:4123 , piece_id=ONBFURPQBWHAFTF5AYGN7WNFXLDCGTMESURQWMLQ33M2ZYWLP36A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.942922918 (308h11m43.814005953s) [noise=true , address=145.79.227.43:4158 , piece_id=F7D6NWXCJADX6RO5BHZ5UDBFXGXJNQ2RDGVCOSXG77SHWLWO62IQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.950468045 (308h11m43.82155109s) [noise=true , address=145.79.227.45:4229 , piece_id=P5U2IXECFYYZV76DWJI2Z4KPKBAUBLCN6LFMDBPDE7N22MSZJTRA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:43.958685012 (308h11m43.829768057s) [noise=true , address=145.79.227.32:4835 , piece_id=V4YYCI67I5EP2ATFZHITELNQQTYDUQJMHPHOKWHDLTATGALUOYTA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.002157492 (308h11m43.873240537s) [noise=true , address=145.79.227.36:4517 , piece_id=ZS5M2VAMRDWTA64BDIWN6AYN7KNGYE7TXIDTQJELP3A7D2SNYK4Q, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.008984552 (308h11m43.880067587s) [noise=true , address=145.79.227.40:4051 , piece_id=JWPLZDTCVTUYIC44PKTFPKPFAGTFY2KQVJ7XZWONKGXBRF6OTJUQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.024468918 (308h11m43.895551963s) [noise=true , address=145.79.227.16:4911 , piece_id=2XGSVKPIZK7S2PQOCXP3DYUUEVYK6VV2KDOEJRCAQINPXTDG4VKQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.029108662 (308h11m43.900191707s) [noise=true , address=145.79.227.33:4869 , piece_id=JFQKKY3YWWR7RAHLQUY5NHJ7Q5KEPQ6BXHK7VNP6MR2ZLQJZG22A, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.113149123 (308h11m43.984232168s) [noise=true , address=77.90.31.10:4520 , piece_id=7TG6PFDCQGTI2PPGC6RAKUOGJ236AG3HRZBENUAF5LP54YTDOYMA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.210967788 (308h11m44.082050833s) [noise=true , address=77.90.31.14:4663 , piece_id=K4JW7L232F3HECA4UGFTKKHN5KBAXYLMGATB2TS76OZOWONK35WA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.211216693 (308h11m44.082299738s) [noise=true , address=45.91.116.7:4011 , piece_id=ADHT66WURRC3O3RK5H6GR66EADQKT6QW6QVRHDRSGQPWI7B4DGZQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224011347 (308h11m44.095094392s) [noise=true , address=77.90.31.21:4880 , piece_id=SIAYRGZ67IYDTOQKYHXFRBNWGU4DCTGD4QLOXZ3THGLCARAWH2KA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: <nil> Dec 18 11:40:44.224032647 (308h11m44.095115682s) [seg_pos={0 0}, upload=270271]: Segment reached optimal threshold of 13 pieces. Dec 18 11:40:44.224252807 (308h11m44.095335852s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224259479 (308h11m44.095342514s) [noise=true , address=45.91.116.9:4262 , piece_id=GP22B6E5EWMN6MHJOJVMY46CVV4JKHU2W3NDBXILJY6CMF27JDOQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224522890 (308h11m44.095605925s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224532509 (308h11m44.095615554s) [noise=true , address=45.91.116.14:4189 , piece_id=LZ7WDOD6Q7IJHCIUMNCBCDGV63RYCBFAZ47FI645IRT6GZQ5P2DA, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224675311 (308h11m44.095758356s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 18 11:40:44.224684007 (308h11m44.095767052s) [noise=true , address=45.91.116.13:4156 , piece_id=CY7XXKATMWKPGUU25MJPSIZ5CRBK3AYJKT7K4J7GVS6LE32G7SLQ, seg_pos={0 0}, upload=270271]: Long tail cancellation due to optimal threshold Dec 18 11:40:44.224702593 (308h11m44.095785638s) [seg_pos={0 0}, upload=270271]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 11:40:44.224782004 (308h11m44.095865039s) [upload=270271]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 11:40:44.224788106 (308h11m44.095871151s) [upload=270271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 11:40:44.516554237 (308h11m44.387637272s) [upload=270271]: Done uploading segments... Dec 18 11:40:44.516596598 (308h11m44.387679633s) [upload=270271]: Upload finished. err: <nil> Duration: 30.002845587s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)