Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.614442529s FAIL (2025-10-25 14:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.614478238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.145250975s FAIL (2025-10-25 14:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.145291163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.250169463s FAIL (2025-10-25 14:37:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.250200601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.05137632s FAIL (2025-10-25 14:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.051405796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.959608386s FAIL (2025-10-25 14:35:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.959641579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.518732301s FAIL (2025-10-25 14:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 7.518775052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.762182168s FAIL (2025-10-25 14:33:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.762222354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.886405285s FAIL (2025-10-25 14:32:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.886428769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.92687313s FAIL (2025-10-25 14:31:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 7.926906241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.885774603s FAIL (2025-10-25 14:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.885865536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.060169165s FAIL (2025-10-25 14:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.060210052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.982797581s FAIL (2025-10-25 14:29:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.982832619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.259579327s FAIL (2025-10-25 14:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 6.259609965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.851503924s FAIL (2025-10-25 14:27:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.851537468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.941830082s FAIL (2025-10-25 14:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.941902099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.811899458s FAIL (2025-10-25 14:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 5.81195293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002037281s FAIL (2025-10-25 14:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ) context deadline exceeded ======LOGS====== Oct 25 14:25:45.590761313 (768h54m31.052853796s) [upload=676323]: Starting upload Oct 25 14:25:45.590860552 (768h54m31.052953025s) [upload=676323]: Done starting upload Oct 25 14:25:45.590979167 (768h54m31.05307165s) [upload=676323]: Uploading segments... Oct 25 14:25:45.591175692 (768h54m31.053268175s) [upload=676323]: Got next segment. Inline: false Oct 25 14:25:45.591242208 (768h54m31.053334681s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 14:25:45.591248821 (768h54m31.053341294s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 14:25:45.847212714 (768h54m31.309305197s) [seg_pos={0 0}, upload=676323]: Begin upload segment... Oct 25 14:25:45.847267218 (768h54m31.309359692s) [seg_pos={0 0}, upload=676323]: Done begin upload segment. Oct 25 14:25:45.847297616 (768h54m31.3093901s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.854593088 (768h54m31.316685571s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855042153 (768h54m31.317134626s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855386869 (768h54m31.317479353s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.855786470 (768h54m31.317878953s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856200278 (768h54m31.318292751s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856520308 (768h54m31.318612781s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.856909950 (768h54m31.319002433s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857332284 (768h54m31.319424767s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.857704994 (768h54m31.319797467s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858092301 (768h54m31.320184784s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858584589 (768h54m31.320677072s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.858960064 (768h54m31.321052547s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859308567 (768h54m31.32140105s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.859646961 (768h54m31.321739444s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:45.860081729 (768h54m31.322174212s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Uploading piece... Oct 25 14:25:47.595358293 (768h54m33.057450776s) [upload=676323]: Next returned nil segment Oct 25 14:25:47.595404610 (768h54m33.057497104s) [upload=676323]: Waiting for error group managing segments... Oct 25 14:25:47.879848215 (768h54m33.341940698s) [noise=true , address=145.79.227.3:4149 , piece_id=HVOFKWZISDZD7WWF2ODWJO3LXTSVIK2NC2UZPJ7LOJTSUPKUTEFA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.913499393 (768h54m33.375591876s) [noise=true , address=145.79.227.7:4295 , piece_id=S35UGZMSWXW4MS3P32PNDQWW6NAVRWNXZIF2RYO2YFNWTJQ6UAKQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.922047680 (768h54m33.384140163s) [noise=true , address=145.79.227.13:4803 , piece_id=2GDBFO73K22E5PU3FUSPG6NQPTK7CDSSMRHBTITDPPXK4J3G56SQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.964828263 (768h54m33.426920736s) [noise=true , address=145.79.227.5:4221 , piece_id=JDGX3B5PUWDQBI6DRBZ2DGZV52RTQOUJKXMWL2KQYJNPIXUXE5ZA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.972897087 (768h54m33.43498956s) [noise=true , address=145.79.227.21:4475 , piece_id=OZGWI5ZM43IU2L5CKP4LHHRCHDQ7MNYIAQWSHVCT53NGMXUCQZUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.974113604 (768h54m33.436206077s) [noise=true , address=145.79.227.18:4005 , piece_id=I5WNEVOHHNAALRKV74GTQ46FAV7ZVJGJZXLL3PC2P5XKKXQUM6VQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.987221834 (768h54m33.449314318s) [noise=true , address=145.79.227.26:4655 , piece_id=S2BSYOCZOJE6AX4L5VB2UQKIGWOTAKOPIX3TG2IDDHLQIYCV2O2Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:47.996650599 (768h54m33.458743082s) [noise=true , address=145.79.227.15:4873 , piece_id=7M7KXKE4UC2QOCJGCLWQA547NDECLMKEAFRVSZME2SZMYZ2TV6NA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.009240222 (768h54m33.471332695s) [noise=true , address=145.79.227.14:4838 , piece_id=77WIXBQ7YYCJSSVFCCZWNS4SU5WTDWKICLMROWVGEINTUG2RARMA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.097108148 (768h54m33.559200621s) [noise=true , address=45.91.116.11:4117 , piece_id=J7DPQCQSUCG2MEFXL3JLD7FHUQGT6O27XFS2USI7JARZGZMERNIQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.113319157 (768h54m33.57541163s) [noise=true , address=45.91.116.4:4939 , piece_id=CSIMXMQA2OQRLO4QHC5KRJPYXE5D5XUM44KH3J7VTSM5ZD355MNA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.130032222 (768h54m33.592124695s) [noise=true , address=77.90.31.12:4592 , piece_id=KG3FVMISPDNZHRFJAZHBXK2JYL3RHCRWXHQJ2PY2LIJ642BKXPQA, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145104081 (768h54m33.607196554s) [noise=true , address=77.90.31.13:4627 , piece_id=TONHUNLZVJCEDQZD7E25Y6IWV5PLBDACJ42XP5F5ABPMN3A3SL4Q, seg_pos={0 0}, upload=676323]: Done uploading piece. err: <nil> Oct 25 14:25:48.145141091 (768h54m33.607233565s) [seg_pos={0 0}, upload=676323]: Segment reached optimal threshold of 13 pieces. Oct 25 14:25:48.145293623 (768h54m33.607386106s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145301127 (768h54m33.60739361s) [noise=true , address=77.90.31.14:4661 , piece_id=XIO54TNSO6RH5DCAG7V4EZWMLQSFJVLOTPFTAI3R5ECVUFM63M2A, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145379746 (768h54m33.607472219s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145386899 (768h54m33.607479373s) [noise=true , address=77.90.31.10:4519 , piece_id=OTZSZJR6NLRRKXIEALOLW6AMUJVUVWCTVZ2BMACNSSRZDQGOTFUQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145401077 (768h54m33.60749355s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Oct 25 14:25:48.145422137 (768h54m33.60751461s) [noise=true , address=77.90.31.21:4877 , piece_id=HCMUMFX2AWD3TV44BM2QQD4PEHBYJULYKHBVZQ7R7M3RZOORUGTQ, seg_pos={0 0}, upload=676323]: Long tail cancellation due to optimal threshold Oct 25 14:25:48.145445261 (768h54m33.607537734s) [seg_pos={0 0}, upload=676323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 14:25:48.145544220 (768h54m33.607636693s) [upload=676323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 14:25:48.145549550 (768h54m33.607642033s) [upload=676323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 14:25:48.438627225 (768h54m33.900719708s) [upload=676323]: Done uploading segments... Oct 25 14:25:48.438651221 (768h54m33.900743704s) [upload=676323]: Upload finished. err: <nil> Duration: 30.002060675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.951766084s FAIL (2025-10-25 14:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.951799566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.093008132s FAIL (2025-10-25 14:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.093043027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.006373316s FAIL (2025-10-25 14:17:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.006404995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.98697965s FAIL (2025-10-25 14:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.98701092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.032316862s FAIL (2025-10-25 14:14:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.03234725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.084925779s FAIL (2025-10-25 14:14:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.084957269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.016635071s FAIL (2025-10-25 14:13:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.016665679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 7.573733419s FAIL (2025-10-25 14:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 7.573758607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.83471432s FAIL (2025-10-25 14:12:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.834763964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.964882027s FAIL (2025-10-25 14:11:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.964917044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.886027962s FAIL (2025-10-25 14:10:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.886053138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.351555504s FAIL (2025-10-25 14:09:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.351588899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.0330241s FAIL (2025-10-25 14:08:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.033061621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.944482488s FAIL (2025-10-25 14:08:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.944509499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.033750894s FAIL (2025-10-25 14:07:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 7.033783357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.892404282s FAIL (2025-10-25 14:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 7.892438587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.018866459s FAIL (2025-10-25 14:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.018899562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.21803662s FAIL (2025-10-25 14:04:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.218085744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.116468127s FAIL (2025-10-25 14:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.116498425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.299791246s FAIL (2025-10-25 14:03:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.299829448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.277554273s FAIL (2025-10-25 14:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.277590433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.850884581s FAIL (2025-10-25 14:01:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.850909788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.944340422s FAIL (2025-10-25 14:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.944380629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.044671011s FAIL (2025-10-25 13:59:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.044706279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.83180348s FAIL (2025-10-25 13:58:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.831837675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.829706286s FAIL (2025-10-25 13:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.829753907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.829195211s FAIL (2025-10-25 13:58:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.829240057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.773922143s FAIL (2025-10-25 13:57:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.773950588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.888583488s FAIL (2025-10-25 13:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.888602705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.335402208s FAIL (2025-10-25 13:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.335436384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.178894969s FAIL (2025-10-25 13:55:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 6.178933052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.944573285s FAIL (2025-10-25 13:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.944605307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.785473208s FAIL (2025-10-25 13:53:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ) context deadline exceeded; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ) context deadline exceeded ======LOGS====== Oct 25 13:31:24.276632063 (768h0m9.738724546s) [upload=675539]: Starting upload Oct 25 13:31:24.276737874 (768h0m9.738830347s) [upload=675539]: Done starting upload Oct 25 13:31:24.276893431 (768h0m9.738985904s) [upload=675539]: Uploading segments... Oct 25 13:31:24.277133076 (768h0m9.739225549s) [upload=675539]: Got next segment. Inline: false Oct 25 13:31:24.277207278 (768h0m9.739299752s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 25 13:31:24.277213260 (768h0m9.739305733s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 25 13:31:24.536702246 (768h0m9.998794719s) [seg_pos={0 0}, upload=675539]: Begin upload segment... Oct 25 13:31:24.536758424 (768h0m9.998850907s) [seg_pos={0 0}, upload=675539]: Done begin upload segment. Oct 25 13:31:24.536782128 (768h0m9.998874611s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537376009 (768h0m9.999468482s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.537767475 (768h0m9.999859958s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538102783 (768h0m10.000195266s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538453110 (768h0m10.000545593s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.538747431 (768h0m10.000839914s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539060808 (768h0m10.001153281s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539323417 (768h0m10.0014159s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.539777882 (768h0m10.001870365s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540101409 (768h0m10.002193892s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540435134 (768h0m10.002527617s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.540739504 (768h0m10.002831977s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541095872 (768h0m10.003188355s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541445087 (768h0m10.00353757s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.541938548 (768h0m10.004031031s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:24.546116724 (768h0m10.008209197s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Uploading piece... Oct 25 13:31:26.167228379 (768h0m11.629320862s) [upload=675539]: Next returned nil segment Oct 25 13:31:26.167251282 (768h0m11.629343765s) [upload=675539]: Waiting for error group managing segments... Oct 25 13:31:26.555861931 (768h0m12.017954404s) [noise=true , address=145.79.227.26:4653 , piece_id=Q64V4OIIBXMTR7S7IN4WOVUSHRUP5KZF7G2MILF67ZYZUFLVZOHA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.569319116 (768h0m12.031411599s) [noise=true , address=145.79.227.13:4802 , piece_id=5IARWG3O6UE2ZYY6CKQRR5RPYOTYRCMFQCIU3AT42RI7WODAFJOQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.571548351 (768h0m12.033640824s) [noise=true , address=145.79.227.14:4838 , piece_id=7GVI4DDIQ7LOEAWC3DZHA6UGK6VAZ3EYEXCU65L2MR3ONTQM4G2A, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.577225531 (768h0m12.039318004s) [noise=true , address=145.79.227.15:4875 , piece_id=6CIW3JOX6YAJFO6DKLSDRL56RNDCCVYP62UQBJVCV3MEDYOE2LSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.578678408 (768h0m12.040770891s) [noise=true , address=145.79.227.18:4006 , piece_id=VHGWA6NUNDTMPA45BPKSTBEG5SGFY3HBQXAQX5F2UMKX3LEWAILA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.587566552 (768h0m12.049659035s) [noise=true , address=145.79.227.25:4619 , piece_id=FK3YL2MF74VJGIN32P4JYJNA7ICDDNXLTGIXTH3S57XMLOUJKMNQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.600627343 (768h0m12.062719816s) [noise=true , address=145.79.227.11:4729 , piece_id=4C6IGV5X2BVPQ7NK7TPHC3XJFAI6TRLDHRPP4I3SPJNHLIZEZAPA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.613983345 (768h0m12.076075828s) [noise=true , address=145.79.227.24:4582 , piece_id=PHSCLJTRLTFH7XJWXTZSEDMJSMYN4VK7ERU53UY6HSWCLYAXZHAA, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.680190256 (768h0m12.142282739s) [noise=true , address=145.79.227.2:4113 , piece_id=RCACR3EB6UBYORNE6R7GN4DIKSBP3YIBGRMIIZFGVQBNY3YMHXFQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.754951385 (768h0m12.217043868s) [noise=true , address=45.91.116.5:4298 , piece_id=UMTDRDRZZAJYMSKRLL72XV4G5WILO7JZTR7BTTSB5XRTOCJZAKJQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.783020404 (768h0m12.245112887s) [noise=true , address=77.90.31.18:4806 , piece_id=JAR5RYZAJA23EOXCITMVJ5KKH4XIIIKA2K6COIV7VRVZPF5DLFSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.836170943 (768h0m12.298263416s) [noise=true , address=45.91.116.9:4263 , piece_id=AROKUCFKNIHHIHVM737KPSEL47KOHHBBGXBAESEGTPSXDMGZOQ2Q, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844864327 (768h0m12.3069568s) [noise=true , address=77.90.31.14:4661 , piece_id=MC5BLN5BIPOH3RW43IFPQM74PBNG4MVH3LGKN4QUEBGOQTMZYBSQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: <nil> Oct 25 13:31:26.844890346 (768h0m12.306982819s) [seg_pos={0 0}, upload=675539]: Segment reached optimal threshold of 13 pieces. Oct 25 13:31:26.845022508 (768h0m12.307114981s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845032567 (768h0m12.30712504s) [noise=true , address=145.79.227.16:4911 , piece_id=BEPWQZB3LTPLRRAU6NEDGICM2VUJTYVKYQVTQE2XJTBJBJMBL2ZQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845141184 (768h0m12.307233667s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845147236 (768h0m12.307239709s) [noise=true , address=77.90.31.18:4807 , piece_id=MGGYYEII7EQY5HEVGEMHAVNSZMS2VVXEHEJCPAE3XKBC4PBT3BPQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845212600 (768h0m12.307305083s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Oct 25 13:31:26.845219533 (768h0m12.307312016s) [noise=true , address=77.90.31.15:4698 , piece_id=BYSGHSDCMX2LDR2FXQMA7R6VUHIHDTZ6IDF5CTW5YESZ5NFMHWLQ, seg_pos={0 0}, upload=675539]: Long tail cancellation due to optimal threshold Oct 25 13:31:26.845234832 (768h0m12.307327305s) [seg_pos={0 0}, upload=675539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 25 13:31:26.845315546 (768h0m12.307408019s) [upload=675539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 25 13:31:26.845322429 (768h0m12.307414902s) [upload=675539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 25 13:31:27.160161012 (768h0m12.622253495s) [upload=675539]: Done uploading segments... Oct 25 13:31:27.160184337 (768h0m12.62227682s) [upload=675539]: Upload finished. err: <nil> Duration: 5.785502524s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)