Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X....X..............X..X..X........X.....................] duration: 6.274496734s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:54:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.274525488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X....X..............X..X..X........X....................] duration: 6.703933686s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.703962702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 20.975766763s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 20.975792351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X....X..............X..X..X........X...................] duration: 18.759346893s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 18.759366641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 42.840206317s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:53:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 42.840258017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 19.846414143s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 19.846438921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...X....X..............X..X..X........X..................] duration: 6.08569009s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:52:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.085716139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 15.444486865s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.444512085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X...X....X..............X..X..X........X.................] duration: 11.693239089s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:51:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 11.693266941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..X...X....X..............X..X..X........X................] duration: 24.760611883s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 24.760639567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 15.136198639s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:51:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.136228227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..X...X....X..............X..X..X........X...............] duration: 6.349640613s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:50:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.349666943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 15.421608934s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.421637276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..X...X....X..............X..X..X........X..............] duration: 6.479167096s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:49:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.479192234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 20.360883011s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 20.360909962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X...X....X..............X..X..X........X.............] duration: 9.05543666s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:48:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 9.055470424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..X...X....X..............X..X..X........X............] duration: 20.478360593s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 20.47839097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 20.638363275s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 20.638391568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..X...X....X..............X..X..X........X...........] duration: 5.954750201s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:46:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 5.954767583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 15.358921902s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.358949564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X..X...X....X..............X..X..X........X..........] duration: 20.654545104s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:46:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 20.654572658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.437683876s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.437710477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X..X...X....X..............X..X..X........X.........] duration: 6.639435713s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:45:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.639455902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 21.078591094s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:44:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 21.078619658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X..X...X....X..............X..X..X........X........] duration: 6.117201081s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:44:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.117234686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 23.705331264s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:43:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 23.705357244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.326288741s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:43:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 22.326315932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004495156s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 21:42:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 1m0.004506477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X..X...X....X..............X..X..X........X.......] duration: 6.240149281s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.240166894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X..X...X....X..............X..X..X........X......] duration: 6.971609857s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:41:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.9716273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X..X...X....X..............X..X..X........X.....] duration: 5.714076162s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:40:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 5.714103865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X............X..X...X....X..............X..X..X........X....] duration: 7.312931858s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:40:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 7.312959861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X............X..X...X....X..............X..X..X........X...] duration: 6.083373844s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 6.083400896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X............X..X...X....X..............X..X..X........X..] duration: 8.660009463s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:38:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 8.660032869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X............X..X...X....X..............X..X..X........X.] duration: 7.168740787s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:38:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 7.168757418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.X............X..X...X....X..............X..X..X........X] duration: 30.0026131s trace: https://jaeger.dev.storj.io/trace/3ff97de69ddc52e2 FAIL (2025-12-29 21:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:37:37.675734039 (582h8m37.546817074s) [upload=510323]: Starting upload Dec 29 21:37:37.675916396 (582h8m37.546999441s) [upload=510323]: Done starting upload Dec 29 21:37:37.675973174 (582h8m37.547056229s) [upload=510323]: Uploading segments... Dec 29 21:37:37.676199785 (582h8m37.54728282s) [upload=510323]: Got next segment. Inline: false Dec 29 21:37:37.676235573 (582h8m37.547318608s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:37:37.676240283 (582h8m37.547323318s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:37:37.924322879 (582h8m37.795405924s) [seg_pos={0 0}, upload=510323]: Begin upload segment... Dec 29 21:37:37.924370339 (582h8m37.795453384s) [seg_pos={0 0}, upload=510323]: Done begin upload segment. Dec 29 21:37:37.924436384 (582h8m37.79551945s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.924448116 (582h8m37.795531152s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925029123 (582h8m37.796112168s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925337310 (582h8m37.796420355s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.925681997 (582h8m37.796765032s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926013607 (582h8m37.797096643s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926423618 (582h8m37.797506663s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.926732286 (582h8m37.797815321s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927356796 (582h8m37.798439831s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.927711300 (582h8m37.798794335s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928141269 (582h8m37.799224314s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928495304 (582h8m37.799578349s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.928955279 (582h8m37.800038314s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.929368957 (582h8m37.800451992s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.930505371 (582h8m37.801588416s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:37.931600166 (582h8m37.802683201s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Uploading piece... Dec 29 21:37:39.625457818 (582h8m39.496540853s) [upload=510323]: Next returned nil segment Dec 29 21:37:39.625472015 (582h8m39.49655506s) [upload=510323]: Waiting for error group managing segments... Dec 29 21:37:39.906405573 (582h8m39.777488619s) [noise=true , address=145.79.227.15:4876 , piece_id=P34U724AFFUWBPQJSLOG4EDL6Z4YOJZPNNB32C7XU5L3IMKRVV5Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.907137227 (582h8m39.778220262s) [noise=true , address=145.79.227.3:4150 , piece_id=VZ6T3FILFEDKKBZZWYAB5OFR2GYVAXU42PDMTFDEAPDMTS2T5YPA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.908342222 (582h8m39.779425257s) [noise=true , address=145.79.227.36:4517 , piece_id=YNPIASXAMXIOKQ7PB6UCCHBZTRM3DB22MSHB42M4F4FJ6TU5PXAA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.910550456 (582h8m39.781633501s) [noise=true , address=145.79.227.50:4373 , piece_id=LZFEKMI5RJTOA3JITS6K7CR4WRHHVH2JZFMXCROFC46K4BDAFMOQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.914536567 (582h8m39.785619612s) [noise=true , address=145.79.227.46:4601 , piece_id=JCEDOYYQT4BLRVR233AXX4N3JMIQXD5W4FPTBY6XHPVKFE3CHSVQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.929342920 (582h8m39.800425955s) [noise=true , address=145.79.227.11:4730 , piece_id=SNK7NEK2U5VXQFXR6USP5T6NACVCIUQ6B24Z6D6VHGT2NZBBB6PA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.933154057 (582h8m39.804237102s) [noise=true , address=145.79.227.26:4655 , piece_id=KEBT7FUV5KJRDLGOU5ZFG5NLM7JL2GRSZGP7A2M7PGVY6JOPFBBA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.938182361 (582h8m39.809265396s) [noise=true , address=145.79.227.45:4230 , piece_id=4E2VJKVSBEQIEUCA5F4EQGX6TCB4D5GMNZSBNLSS6UOKUTQ2R63Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:39.948171021 (582h8m39.819254066s) [noise=true , address=145.79.227.24:4583 , piece_id=AEPKLJQDRBT5WDDB4X7MGDYNTL5RGCTQLUVXOX3DZHWN636RGAZQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.078604556 (582h8m39.949687591s) [noise=true , address=77.90.31.13:4627 , piece_id=NH7YKD67X574AZCQMHKRTWPB6SEIVUI53Q4NNWXOVRJEIHL34Z4A, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.079147720 (582h8m39.950230755s) [noise=true , address=77.90.31.11:4553 , piece_id=AKTQIHX4QFHEOICZOU7LQDE4WLSLRRFZHADIP7D5KBYJJTCDK6KA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.096835040 (582h8m39.967918095s) [noise=true , address=77.90.31.16:4735 , piece_id=VIKUUJ6MWAKTMASBJPDUHHREAA5JW6OEXKVIEZTGQI7TJSSMI3DQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178520742 (582h8m40.049603787s) [noise=true , address=45.91.116.5:4298 , piece_id=2FU4TFKG5AWJDUU5UU3UKR7IDYK5LCHPOJCEUTUCLQXYR3O7UOCQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: <nil> Dec 29 21:37:40.178548394 (582h8m40.049631429s) [seg_pos={0 0}, upload=510323]: Segment reached optimal threshold of 13 pieces. Dec 29 21:37:40.178881869 (582h8m40.049964904s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178925863 (582h8m40.050008898s) [noise=true , address=45.91.116.13:4155 , piece_id=PDE2MTIB7QCHGICCB4YREAG7JTMAYMWQOOGTY36XLIEV75IBH5QA, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.178955710 (582h8m40.050038745s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 29 21:37:40.178965257 (582h8m40.050048292s) [noise=true , address=45.91.116.9:4262 , piece_id=7MQLY6YPNJD6KBBIHQAZYIYNJJY5GU55ZQSBVQDOYVWITERKCZ4Q, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179143908 (582h8m40.050226944s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35452->45.91.116.7:4009: use of closed network connection Dec 29 21:37:40.179163786 (582h8m40.050246821s) [noise=true , address=45.91.116.7:4009 , piece_id=SQ6T33SVO326V5DY4F7K5MSPZCOCTF37E7BZJXDPUU3L57VUGTAQ, seg_pos={0 0}, upload=510323]: Long tail cancellation due to optimal threshold Dec 29 21:37:40.179184556 (582h8m40.050267591s) [seg_pos={0 0}, upload=510323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:37:40.179270499 (582h8m40.050353544s) [upload=510323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:37:40.179277211 (582h8m40.050360247s) [upload=510323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:37:40.476789228 (582h8m40.347872273s) [upload=510323]: Done uploading segments... Dec 29 21:37:40.476926109 (582h8m40.348009164s) [upload=510323]: Upload finished. err: <nil> Duration: 30.002637487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.X............X..X...X....X..............X..X..X........] duration: 6.393877264s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:36:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 6.393905298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X............X..X...X....X..............X..X..X.......] duration: 29.363572728s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:36:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 29.363587596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.X............X..X...X....X..............X..X..X......] duration: 28.49098644s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:36:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 28.491015085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.X............X..X...X....X..............X..X..X.....] duration: 30.002750133s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 30.002775161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X.X............X..X...X....X..............X..X..X....] duration: 6.195687403s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:34:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 6.195713273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X.X............X..X...X....X..............X..X..X...] duration: 5.984879683s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:33:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 5.984907275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X.X............X..X...X....X..............X..X..X..] duration: 6.169737326s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:33:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 6.169767824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X.X............X..X...X....X..............X..X..X.] duration: 6.173825115s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:31:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 6.173849051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X.X............X..X...X....X..............X..X..X] duration: 30.003278252s trace: https://jaeger.dev.storj.io/trace/12fa8a0650e15a9b FAIL (2025-12-29 21:31:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 21:31:22.875121659 (582h2m22.746204704s) [upload=510233]: Starting upload Dec 29 21:31:22.875209666 (582h2m22.746292701s) [upload=510233]: Done starting upload Dec 29 21:31:22.875269060 (582h2m22.746352095s) [upload=510233]: Uploading segments... Dec 29 21:31:22.875432812 (582h2m22.746515857s) [upload=510233]: Got next segment. Inline: false Dec 29 21:31:22.875499929 (582h2m22.746582974s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:31:22.875505690 (582h2m22.746588735s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:31:23.126266997 (582h2m22.997350032s) [seg_pos={0 0}, upload=510233]: Begin upload segment... Dec 29 21:31:23.126312614 (582h2m22.997395649s) [seg_pos={0 0}, upload=510233]: Done begin upload segment. Dec 29 21:31:23.126338273 (582h2m22.997421308s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.126467459 (582h2m22.997550494s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127232585 (582h2m22.998315621s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127519872 (582h2m22.998602907s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127712500 (582h2m22.998795535s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.127897802 (582h2m22.998980837s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128161465 (582h2m22.9992445s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128267536 (582h2m22.999350572s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128610550 (582h2m22.999693585s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128686735 (582h2m22.99976977s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.128934346 (582h2m23.000017381s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129237684 (582h2m23.000320719s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129255417 (582h2m23.000338452s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129550920 (582h2m23.000633955s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.129769366 (582h2m23.000852411s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:23.130310497 (582h2m23.001393532s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Uploading piece... Dec 29 21:31:24.820517311 (582h2m24.691600346s) [upload=510233]: Next returned nil segment Dec 29 21:31:24.820531849 (582h2m24.691614894s) [upload=510233]: Waiting for error group managing segments... Dec 29 21:31:25.098271657 (582h2m24.969354702s) [noise=true , address=145.79.227.4:4186 , piece_id=EG4HBOMM745ZLU376KM54Q45H2EGPP73KPEPLUCXJG6USNZWF5IQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.106849981 (582h2m24.977933016s) [noise=true , address=145.79.227.15:4874 , piece_id=TOMUAPAU2UZ5KHTMPLNW2T74NIMUBYNJZUSPZCTB7Q3RIPLUVZXA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.116403944 (582h2m24.987486989s) [noise=true , address=145.79.227.39:4013 , piece_id=QQBCSNCD3PVFWC3OHXJAKOMOH2EWNQS6N3ZGRU3L6KZTVLWLP5MQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.130318198 (582h2m25.001401253s) [noise=true , address=145.79.227.2:4114 , piece_id=75WWVWWQMVYSRTSHJTZXXISOPXHASER7CUIOSXCLLQLRIPHLGSIA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.131401451 (582h2m25.002484486s) [noise=true , address=145.79.227.6:4257 , piece_id=B7H2YY7UGJSIZQXR3JFUDR3FXIMQPATDSPPC3NYGYHEFVG6YHCIQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.133500258 (582h2m25.004583303s) [noise=true , address=145.79.227.17:4971 , piece_id=E5IG4MB7TGNVIV6JZQ3FAM6JLDI3JI6ZXIBXG4XRHQJVIT5GXNYA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.135573916 (582h2m25.006656961s) [noise=true , address=145.79.227.14:4837 , piece_id=GHQ75EKWAULFC3MJTX37EE7GHNL2KN6OZSSMQ2FISPXD6BL3POAQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.139269623 (582h2m25.010352668s) [noise=true , address=145.79.227.51:4411 , piece_id=3BGNER5RUGYYQJ4UPJ7A46R35RCRUCK3QEQVAXLBEWV54ECMOF4Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.148496102 (582h2m25.019579147s) [noise=true , address=145.79.227.18:4006 , piece_id=AFLEOYSOWQPFC6QS6LS4PLWURKWK4U3AR6K54ZKXATRXKPQXEWYQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.277750582 (582h2m25.148833627s) [noise=true , address=77.90.31.18:4808 , piece_id=I6QNJ4QBYA5VWCFYRI3CQNHLWJBWDNTKCG2G4JLUIHM5BALVSLGQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.365436960 (582h2m25.236519995s) [noise=true , address=45.91.116.4:4940 , piece_id=BP2UBBPW4PRSJCPKM6B7O2ZQ2GT24L2DUSICOTBPF4ZUKNYROJ3Q, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.373857244 (582h2m25.244940279s) [noise=true , address=45.91.116.6:4973 , piece_id=MFK6WGV3DSTNPTXFB5PZEG5EAITDLJULM65OAB5OVTGPUNTUVHNQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382385141 (582h2m25.253468186s) [noise=true , address=45.91.116.12:4225 , piece_id=J2Y5RJ3W2OGNKLEKYYNVS47FTNAS6YNXI5VAZA27OAVNKWVR2TPA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: <nil> Dec 29 21:31:25.382400691 (582h2m25.253483726s) [seg_pos={0 0}, upload=510233]: Segment reached optimal threshold of 13 pieces. Dec 29 21:31:25.382626531 (582h2m25.253709566s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40888->145.79.227.48:4301: use of closed network connection Dec 29 21:31:25.382634397 (582h2m25.253717432s) [noise=true , address=145.79.227.48:4301 , piece_id=DBDGSYFLGMOUI6XEQNQXXVIYGGJHI34XIR7RZRDXUX62U7S3YYAA, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382682909 (582h2m25.253765944s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382690202 (582h2m25.253773247s) [noise=true , address=45.91.116.10:4082 , piece_id=422F6LGH4N6L3CVPXVEDRIQ6PBIJKYPT2OPPFQY7GN6YDP2KMUHQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382740298 (582h2m25.253823343s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 21:31:25.382745137 (582h2m25.253828172s) [noise=true , address=145.79.227.44:4195 , piece_id=BFCNBQZWWWKFZZ5Z74EQTVL63OLYMMQCIEHEIGT26P67TXU47QWQ, seg_pos={0 0}, upload=510233]: Long tail cancellation due to optimal threshold Dec 29 21:31:25.382757681 (582h2m25.253840716s) [seg_pos={0 0}, upload=510233]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:31:25.382836160 (582h2m25.253919205s) [upload=510233]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:31:25.382840999 (582h2m25.253924034s) [upload=510233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:31:25.687461133 (582h2m25.558544178s) [upload=510233]: Done uploading segments... Dec 29 21:31:25.687511688 (582h2m25.558594724s) [upload=510233]: Upload finished. err: <nil> Duration: 30.003298421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.X............X..X...X....X..............X..X..] duration: 20.323389114s trace: https://jaeger.dev.storj.io/trace/6c7db33f43e01c0c FAIL (2025-12-29 21:31:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m34.878748919s) [upload=510206]: Uploading segments... Dec 29 21:29:35.007790541 (582h0m34.878873586s) [upload=510206]: Got next segment. Inline: false Dec 29 21:29:35.007830938 (582h0m34.878913973s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:29:35.007835957 (582h0m34.878919012s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:29:35.255825105 (582h0m35.12690816s) [seg_pos={0 0}, upload=510206]: Begin upload segment... Dec 29 21:29:35.255877395 (582h0m35.12696044s) [seg_pos={0 0}, upload=510206]: Done begin upload segment. Dec 29 21:29:35.255917431 (582h0m35.127000466s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256349544 (582h0m35.127432579s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256942142 (582h0m35.128025177s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257266931 (582h0m35.128349976s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257278423 (582h0m35.128361458s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257320362 (582h0m35.128403398s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257597520 (582h0m35.128680555s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257746103 (582h0m35.128829168s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258267716 (582h0m35.129350761s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258692085 (582h0m35.12977513s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259213397 (582h0m35.130296442s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259716195 (582h0m35.13079923s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.264620403 (582h0m35.135703438s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.271076757 (582h0m35.142159802s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.273119908 (582h0m35.144202953s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.279628280 (582h0m35.150711325s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:36.965904552 (582h0m36.836987587s) [upload=510206]: Next returned nil segment Dec 29 21:29:36.965921194 (582h0m36.837004239s) [upload=510206]: Waiting for error group managing segments... Dec 29 21:29:37.247790424 (582h0m37.118873459s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.258512842 (582h0m37.129595877s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.261574432 (582h0m37.132657467s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.264519449 (582h0m37.135602484s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.286808831 (582h0m37.157891866s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.428419409 (582h0m37.299502454s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.435533696 (582h0m37.306616731s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.492779711 (582h0m37.363862746s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.508922730 (582h0m37.380005775s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.513197219 (582h0m37.384280254s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.541130257 (582h0m37.412213292s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.215703515 (582h0m38.08678656s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328222208 (582h0m38.199305243s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328240051 (582h0m38.199323086s) [seg_pos={0 0}, upload=510206]: Segment reached optimal threshold of 13 pieces. Dec 29 21:29:38.328487322 (582h0m38.199570357s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 21:29:38.328496189 (582h0m38.199579234s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328598544 (582h0m38.199681579s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection Dec 29 21:29:38.328613673 (582h0m38.199696708s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328747859 (582h0m38.199830904s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection Dec 29 21:29:38.328760422 (582h0m38.199843467s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328785640 (582h0m38.199868686s) [seg_pos={0 0}, upload=510206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:29:38.328894959 (582h0m38.199978004s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:29:38.328902853 (582h0m38.199985888s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:29:38.631889575 (582h0m38.50297261s) [upload=510206]: Done uploading segments... Dec 29 21:29:38.631910284 (582h0m38.502993329s) [upload=510206]: Upload finished. err: <nil> Duration: 20.323416206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.X............X..X...X....X..............X..X.] duration: 27.933620479s trace: https://jaeger.dev.storj.io/trace/6c7db33f43e01c0c FAIL (2025-12-29 21:30:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m34.878748919s) [upload=510206]: Uploading segments... Dec 29 21:29:35.007790541 (582h0m34.878873586s) [upload=510206]: Got next segment. Inline: false Dec 29 21:29:35.007830938 (582h0m34.878913973s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:29:35.007835957 (582h0m34.878919012s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:29:35.255825105 (582h0m35.12690816s) [seg_pos={0 0}, upload=510206]: Begin upload segment... Dec 29 21:29:35.255877395 (582h0m35.12696044s) [seg_pos={0 0}, upload=510206]: Done begin upload segment. Dec 29 21:29:35.255917431 (582h0m35.127000466s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256349544 (582h0m35.127432579s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256942142 (582h0m35.128025177s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257266931 (582h0m35.128349976s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257278423 (582h0m35.128361458s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257320362 (582h0m35.128403398s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257597520 (582h0m35.128680555s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257746103 (582h0m35.128829168s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258267716 (582h0m35.129350761s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258692085 (582h0m35.12977513s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259213397 (582h0m35.130296442s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259716195 (582h0m35.13079923s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.264620403 (582h0m35.135703438s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.271076757 (582h0m35.142159802s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.273119908 (582h0m35.144202953s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.279628280 (582h0m35.150711325s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:36.965904552 (582h0m36.836987587s) [upload=510206]: Next returned nil segment Dec 29 21:29:36.965921194 (582h0m36.837004239s) [upload=510206]: Waiting for error group managing segments... Dec 29 21:29:37.247790424 (582h0m37.118873459s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.258512842 (582h0m37.129595877s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.261574432 (582h0m37.132657467s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.264519449 (582h0m37.135602484s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.286808831 (582h0m37.157891866s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.428419409 (582h0m37.299502454s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.435533696 (582h0m37.306616731s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.492779711 (582h0m37.363862746s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.508922730 (582h0m37.380005775s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.513197219 (582h0m37.384280254s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.541130257 (582h0m37.412213292s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.215703515 (582h0m38.08678656s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328222208 (582h0m38.199305243s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328240051 (582h0m38.199323086s) [seg_pos={0 0}, upload=510206]: Segment reached optimal threshold of 13 pieces. Dec 29 21:29:38.328487322 (582h0m38.199570357s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 21:29:38.328496189 (582h0m38.199579234s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328598544 (582h0m38.199681579s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection Dec 29 21:29:38.328613673 (582h0m38.199696708s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328747859 (582h0m38.199830904s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection Dec 29 21:29:38.328760422 (582h0m38.199843467s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328785640 (582h0m38.199868686s) [seg_pos={0 0}, upload=510206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:29:38.328894959 (582h0m38.199978004s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:29:38.328902853 (582h0m38.199985888s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:29:38.631889575 (582h0m38.50297261s) [upload=510206]: Done uploading segments... Dec 29 21:29:38.631910284 (582h0m38.502993329s) [upload=510206]: Upload finished. err: <nil> Duration: 27.933646829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.X............X..X...X....X..............X..X] duration: 30.003182697s trace: https://jaeger.dev.storj.io/trace/6c7db33f43e01c0c FAIL (2025-12-29 21:30:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m34.878748919s) [upload=510206]: Uploading segments... Dec 29 21:29:35.007790541 (582h0m34.878873586s) [upload=510206]: Got next segment. Inline: false Dec 29 21:29:35.007830938 (582h0m34.878913973s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:29:35.007835957 (582h0m34.878919012s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:29:35.255825105 (582h0m35.12690816s) [seg_pos={0 0}, upload=510206]: Begin upload segment... Dec 29 21:29:35.255877395 (582h0m35.12696044s) [seg_pos={0 0}, upload=510206]: Done begin upload segment. Dec 29 21:29:35.255917431 (582h0m35.127000466s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256349544 (582h0m35.127432579s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.256942142 (582h0m35.128025177s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257266931 (582h0m35.128349976s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257278423 (582h0m35.128361458s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257320362 (582h0m35.128403398s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257597520 (582h0m35.128680555s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.257746103 (582h0m35.128829168s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258267716 (582h0m35.129350761s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.258692085 (582h0m35.12977513s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259213397 (582h0m35.130296442s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.259716195 (582h0m35.13079923s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.264620403 (582h0m35.135703438s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.271076757 (582h0m35.142159802s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.273119908 (582h0m35.144202953s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:35.279628280 (582h0m35.150711325s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Uploading piece... Dec 29 21:29:36.965904552 (582h0m36.836987587s) [upload=510206]: Next returned nil segment Dec 29 21:29:36.965921194 (582h0m36.837004239s) [upload=510206]: Waiting for error group managing segments... Dec 29 21:29:37.247790424 (582h0m37.118873459s) [noise=true , address=145.79.227.2:4114 , piece_id=DJZ5WIJVBTRFI5V7RTQ2L5FP55QD26X6HUEQGBAPAIOYLZE34Q6A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.258512842 (582h0m37.129595877s) [noise=true , address=145.79.227.47:4266 , piece_id=JUZX2HLH4G3Y2V65UCJWIHPCFY6IZUR7J76QYBIJLWQGJQ4YTTFQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.261574432 (582h0m37.132657467s) [noise=true , address=145.79.227.38:4977 , piece_id=URCH2HOVNJL2XUDG7HWYGNID7V2Z7PO44ONP6AJMLLYF7NKGY5YA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.264519449 (582h0m37.135602484s) [noise=true , address=145.79.227.39:4013 , piece_id=SECFNL5EQBAOOW3FE6FCGG3U7VVALC2LT5S757J3REXIMMWGS62Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.286808831 (582h0m37.157891866s) [noise=true , address=145.79.227.46:4603 , piece_id=2GMZVGHEUK7R2HIM2BVPGW327MRDUORGZWABY5TXJYX35CW2HZIQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.428419409 (582h0m37.299502454s) [noise=true , address=77.90.31.12:4590 , piece_id=OB3A24WW5DREKVA3FLYCFULSUOHI3R2RDNTA2R6IQYNKUT3J2TOQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.435533696 (582h0m37.306616731s) [noise=true , address=77.90.31.16:4736 , piece_id=UMI24TPQAGFE7OJ2KZTLBWXBHGN5RO5GH3S6HZXVI75TEJGOVO7Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.492779711 (582h0m37.363862746s) [noise=true , address=45.91.116.5:4298 , piece_id=ME2OET5YSFZI2W7UL675JPDR62GTTLBSKIWZQAYU7ZPMBFSVM3EQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.508922730 (582h0m37.380005775s) [noise=true , address=45.91.116.6:4976 , piece_id=Z7LH3S56ZNUZEHMOYIQP26FO5AHKHGJD4CLCA3UI7HRYLIMNSBNA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.513197219 (582h0m37.384280254s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSGG4IJ5E6DPAQVAS7IQRK5YBCVZZB34DEL65QS3XVESJQQUG33Q, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:37.541130257 (582h0m37.412213292s) [noise=true , address=45.91.116.4:4940 , piece_id=MOPTVLZXD5VA7IIQ25LQ4BJOFCTMKA3VDM3I3SFQ4S6R5NW5NPLQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.215703515 (582h0m38.08678656s) [noise=true , address=45.91.116.9:4262 , piece_id=IMGB42FYHUQGUX5QR5SCL5FLGL4W76MVHXHYP3NJMB2BCVPWKYLA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328222208 (582h0m38.199305243s) [noise=true , address=45.91.116.13:4155 , piece_id=BKMLYQLFXGFYIVUYPIFLB76LOHNPZSFCNQRUEYJTI3CEL4XVY6ZQ, seg_pos={0 0}, upload=510206]: Done uploading piece. err: <nil> Dec 29 21:29:38.328240051 (582h0m38.199323086s) [seg_pos={0 0}, upload=510206]: Segment reached optimal threshold of 13 pieces. Dec 29 21:29:38.328487322 (582h0m38.199570357s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 21:29:38.328496189 (582h0m38.199579234s) [noise=true , address=77.90.31.10:4520 , piece_id=SAQGUWMWRZ5E7F6M5EBWTVU6SHPLVCIQRI5ZPOSUG4OASAFM3VEA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328598544 (582h0m38.199681579s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52098->145.79.227.4:4186: use of closed network connection Dec 29 21:29:38.328613673 (582h0m38.199696708s) [noise=true , address=145.79.227.4:4186 , piece_id=SIMXVSP5FBINBOQFDL6XM2EZTA2K3RPKHUZZNBJKFF2VH4TVN4HA, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328747859 (582h0m38.199830904s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46328->145.79.227.16:4912: use of closed network connection Dec 29 21:29:38.328760422 (582h0m38.199843467s) [noise=true , address=145.79.227.16:4912 , piece_id=UO52PPHPO5TOZB6SEVOCVULX3YTRZTSKHHUAVWWAVJNI5W4VMF3A, seg_pos={0 0}, upload=510206]: Long tail cancellation due to optimal threshold Dec 29 21:29:38.328785640 (582h0m38.199868686s) [seg_pos={0 0}, upload=510206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:29:38.328894959 (582h0m38.199978004s) [upload=510206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:29:38.328902853 (582h0m38.199985888s) [upload=510206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:29:38.631889575 (582h0m38.50297261s) [upload=510206]: Done uploading segments... Dec 29 21:29:38.631910284 (582h0m38.502993329s) [upload=510206]: Upload finished. err: <nil> Duration: 30.0032052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X.X............X..X...X....X..............X..] duration: 11.26186159s trace: https://jaeger.dev.storj.io/trace/6c40549e61347233 FAIL (2025-12-29 21:29:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 2FKKHALYS4VCT52T6USOB4PL4XOYWUM3PJG4AT63EFLFGBDAWMWA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: MHSTE3Z2G4HFRFTYKKGPT5UMZJ5BSBBQBD2OHO4OGN3UIBU6L2VQ) context deadline exceeded ======LOGS====== Dec 29 21:27:08.615627098 (581h58m8.486710153s) [upload=510171]: Starting upload Dec 29 21:27:08.615714866 (581h58m8.486797931s) [upload=510171]: Done starting upload Dec 29 21:27:08.615754601 (581h58m8.486837636s) [upload=510171]: Uploading segments... Dec 29 21:27:08.615977015 (581h58m8.48706005s) [upload=510171]: Got next segment. Inline: false Dec 29 21:27:08.616037710 (581h58m8.487120745s) [upload=510171]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 21:27:08.616074290 (581h58m8.487157365s) [upload=510171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 21:27:08.861356464 (581h58m8.732439519s) [seg_pos={0 0}, upload=510171]: Begin upload segment... Dec 29 21:27:08.861415566 (581h58m8.732498611s) [seg_pos={0 0}, upload=510171]: Done begin upload segment. Dec 29 21:27:08.861460873 (581h58m8.732543918s) [noise=true , address=145.79.227.15:4876 , piece_id=QMKGLN655AD4XKI6ZBTQJ24RXHXJHOR5BZSVAOM2XIMFSO5VM5OA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.861468528 (581h58m8.732551563s) [noise=true , address=45.91.116.12:4228 , piece_id=AQCXI2IKK37NSQWPPBNTJNANJTPUAJEXGUMQUD7PMIS6GN7WBNKQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862129606 (581h58m8.733212641s) [noise=true , address=145.79.227.11:4731 , piece_id=VLHWSUYDUEXBWXLUZR2O4MZ42Z6SZZODH4FTIMFCCGUW7RUKUM5Q, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862192015 (581h58m8.73327505s) [noise=true , address=45.91.116.11:4119 , piece_id=MHSTE3Z2G4HFRFTYKKGPT5UMZJ5BSBBQBD2OHO4OGN3UIBU6L2VQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862501354 (581h58m8.733584389s) [noise=true , address=145.79.227.4:4185 , piece_id=AUPOKFJZQXIZW2QOVN5CPWICUJGZCC4K74NJXS3M4LQYUHMVXKAQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862539197 (581h58m8.733622232s) [noise=true , address=45.91.116.10:4083 , piece_id=VU7YIOMQCZLJWDC3DP43MV7YKD4HDJ6ADVHJBP3FYKABLRRRDDJQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862811194 (581h58m8.733894229s) [noise=true , address=145.79.227.46:4602 , piece_id=LRV5LDDB6DOYWGFQRGDLAY2ZBBRW43XQOPPI6E46SI72CKNAJ6SQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.862915663 (581h58m8.733998698s) [noise=true , address=145.79.227.39:4015 , piece_id=CGGCLZM55YYBRKNYGPIDVBSGIB4XSJ43XQTT25CYUOLKBSRLFESQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.863194043 (581h58m8.734277078s) [noise=true , address=145.79.227.24:4583 , piece_id=H6RIH4GIYG2JPMY563S4L3VEN7KE7XVR74C72R3WKCJ2K3E3TXTA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.863380779 (581h58m8.734463814s) [noise=true , address=145.79.227.51:4411 , piece_id=62U7LZMTAOSQHDHILM52JOC3FZ34WN35QUPS34CSISENI5BO2BIA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.863507259 (581h58m8.734590304s) [noise=true , address=45.91.116.8:4047 , piece_id=GJAC3HMEDRIXG2NNKK2RVX77R5UVFJNZKUL7QNQNLIVJUH2GPD2Q, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.863819924 (581h58m8.73490297s) [noise=true , address=77.90.31.11:4555 , piece_id=TLPAAEZCKLDDCXWSAWYAB7I7HUVHATWHMSLGAHPEX5T3CUMRY4UA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.863861283 (581h58m8.734944328s) [noise=true , address=145.79.227.3:4149 , piece_id=2IINR32HSWN5BGA3DRLBWKCBA7R74J3BX4LTKT64Z76B2JDY2Y2A, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.864209376 (581h58m8.735292421s) [noise=true , address=77.90.31.14:4662 , piece_id=EKHXEVF3GR36K2EQZCOQJDWLNUKQ72TNUOJXOBHB33PFNOH7FDGA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.864233502 (581h58m8.735316537s) [noise=true , address=77.90.31.21:4878 , piece_id=2FKKHALYS4VCT52T6USOB4PL4XOYWUM3PJG4AT63EFLFGBDAWMWA, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:08.864596564 (581h58m8.735679599s) [noise=true , address=77.90.31.10:4517 , piece_id=A2R4OC4GJ74XFAJQUE7JM3ZVUMUQZTK7UR6EUHX65Z4L7JIM42IQ, seg_pos={0 0}, upload=510171]: Uploading piece... Dec 29 21:27:10.568949075 (581h58m10.44003212s) [upload=510171]: Next returned nil segment Dec 29 21:27:10.568964314 (581h58m10.440047359s) [upload=510171]: Waiting for error group managing segments... Dec 29 21:27:10.846235690 (581h58m10.717318735s) [noise=true , address=145.79.227.39:4015 , piece_id=CGGCLZM55YYBRKNYGPIDVBSGIB4XSJ43XQTT25CYUOLKBSRLFESQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.850377306 (581h58m10.721460341s) [noise=true , address=145.79.227.46:4602 , piece_id=LRV5LDDB6DOYWGFQRGDLAY2ZBBRW43XQOPPI6E46SI72CKNAJ6SQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.862269871 (581h58m10.733352916s) [noise=true , address=145.79.227.15:4876 , piece_id=QMKGLN655AD4XKI6ZBTQJ24RXHXJHOR5BZSVAOM2XIMFSO5VM5OA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.863838438 (581h58m10.734921483s) [noise=true , address=145.79.227.3:4149 , piece_id=2IINR32HSWN5BGA3DRLBWKCBA7R74J3BX4LTKT64Z76B2JDY2Y2A, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.868028226 (581h58m10.739111261s) [noise=true , address=145.79.227.51:4411 , piece_id=62U7LZMTAOSQHDHILM52JOC3FZ34WN35QUPS34CSISENI5BO2BIA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.868823580 (581h58m10.739906615s) [noise=true , address=145.79.227.11:4731 , piece_id=VLHWSUYDUEXBWXLUZR2O4MZ42Z6SZZODH4FTIMFCCGUW7RUKUM5Q, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:10.870310372 (581h58m10.741393407s) [noise=true , address=145.79.227.24:4583 , piece_id=H6RIH4GIYG2JPMY563S4L3VEN7KE7XVR74C72R3WKCJ2K3E3TXTA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.033945337 (581h58m10.905028382s) [noise=true , address=77.90.31.10:4517 , piece_id=A2R4OC4GJ74XFAJQUE7JM3ZVUMUQZTK7UR6EUHX65Z4L7JIM42IQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.037416186 (581h58m10.908499221s) [noise=true , address=77.90.31.21:4878 , piece_id=2FKKHALYS4VCT52T6USOB4PL4XOYWUM3PJG4AT63EFLFGBDAWMWA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.042154499 (581h58m10.913237534s) [noise=true , address=77.90.31.14:4662 , piece_id=EKHXEVF3GR36K2EQZCOQJDWLNUKQ72TNUOJXOBHB33PFNOH7FDGA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.090557786 (581h58m10.961640831s) [noise=true , address=45.91.116.8:4047 , piece_id=GJAC3HMEDRIXG2NNKK2RVX77R5UVFJNZKUL7QNQNLIVJUH2GPD2Q, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.107942659 (581h58m10.979025694s) [noise=true , address=45.91.116.11:4119 , piece_id=MHSTE3Z2G4HFRFTYKKGPT5UMZJ5BSBBQBD2OHO4OGN3UIBU6L2VQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.117593506 (581h58m10.988676541s) [noise=true , address=45.91.116.10:4083 , piece_id=VU7YIOMQCZLJWDC3DP43MV7YKD4HDJ6ADVHJBP3FYKABLRRRDDJQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: <nil> Dec 29 21:27:11.117613484 (581h58m10.988696529s) [seg_pos={0 0}, upload=510171]: Segment reached optimal threshold of 13 pieces. Dec 29 21:27:11.117780492 (581h58m10.988863527s) [noise=true , address=145.79.227.4:4185 , piece_id=AUPOKFJZQXIZW2QOVN5CPWICUJGZCC4K74NJXS3M4LQYUHMVXKAQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:35740->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:35740->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->145.79.227.4:4185: use of closed network connection Dec 29 21:27:11.117786744 (581h58m10.988869779s) [noise=true , address=145.79.227.4:4185 , piece_id=AUPOKFJZQXIZW2QOVN5CPWICUJGZCC4K74NJXS3M4LQYUHMVXKAQ, seg_pos={0 0}, upload=510171]: Long tail cancellation due to optimal threshold Dec 29 21:27:11.117913275 (581h58m10.98899632s) [noise=true , address=45.91.116.12:4228 , piece_id=AQCXI2IKK37NSQWPPBNTJNANJTPUAJEXGUMQUD7PMIS6GN7WBNKQ, seg_pos={0 0}, upload=510171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 21:27:11.117923595 (581h58m10.98900665s) [noise=true , address=45.91.116.12:4228 , piece_id=AQCXI2IKK37NSQWPPBNTJNANJTPUAJEXGUMQUD7PMIS6GN7WBNKQ, seg_pos={0 0}, upload=510171]: Long tail cancellation due to optimal threshold Dec 29 21:27:11.118106623 (581h58m10.989189668s) [noise=true , address=77.90.31.11:4555 , piece_id=TLPAAEZCKLDDCXWSAWYAB7I7HUVHATWHMSLGAHPEX5T3CUMRY4UA, seg_pos={0 0}, upload=510171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 29 21:27:11.118114829 (581h58m10.989197874s) [noise=true , address=77.90.31.11:4555 , piece_id=TLPAAEZCKLDDCXWSAWYAB7I7HUVHATWHMSLGAHPEX5T3CUMRY4UA, seg_pos={0 0}, upload=510171]: Long tail cancellation due to optimal threshold Dec 29 21:27:11.118138404 (581h58m10.989221449s) [seg_pos={0 0}, upload=510171]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 21:27:11.118242843 (581h58m10.989325888s) [upload=510171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:27:11.118250437 (581h58m10.989333492s) [upload=510171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:27:11.406026175 (581h58m11.27710923s) [upload=510171]: Done uploading segments... Dec 29 21:27:11.406049570 (581h58m11.277132605s) [upload=510171]: Upload finished. err: <nil> Duration: 11.261877851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 26.105807751s trace: https://jaeger.dev.storj.io/trace/13e148e8c086bc6f FAIL (2025-12-29 21:28:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, Piece ID: CTFFZF4I7CJ6GOV2BBQHU6LNKKN43CUTYC5EADK5V2CDRWLBHKOA) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: LPEOROV7YMHMKHC4E3EHOY6Y6CELM3EDXTSVY6MNK3AEJ2OCWXRQ) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: T5LUJEOIVHDT7S6YQAPQNJR5XTZQ47KRD6KFAHPRI7Q76QEF22RA) context deadline exceeded ======LOGS====== 3 , piece_id=5YHXRO4AANIHGG4NTCQG32X36EO3BVNMDOGUV7SUPCPK25QPIVEQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.320961123 (581h14m21.192044158s) [noise=true , address=145.79.227.40:4073 , piece_id=QU5OVXFRPC5QJQEWDO5IDDZPTB4BM2YRYBV2WBZXHRWU5RO7PPFA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.321361365 (581h14m21.1924444s) [noise=true , address=77.90.31.21:4883 , piece_id=GAMV3JW2MMG4NNVXIIWWKLSNLBJM332XYZILI3CNKHS5JPVTYCSQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.321374290 (581h14m21.192457325s) [noise=true , address=145.79.227.34:4938 , piece_id=6ETRVJX5B2SIHOHNREBMA3MSXG42JPCZ6CF73YW7OULR2J4KWMUQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.321649955 (581h14m21.19273299s) [noise=true , address=145.79.227.35:4493 , piece_id=QPUTA52LHETXXVXWOZHXH4UM7JVWHUB63LGQSAI76GTOYFITNNSQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.321742621 (581h14m21.192825666s) [noise=true , address=64.49.13.140:4511 , piece_id=FX3ROLTTGBGY3PHEDFAW3I4ZS4IWWPIWHZJGS6GLFQGMUZJNTRGQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.322277520 (581h14m21.193360565s) [noise=true , address=145.79.227.7:4322 , piece_id=3ZP5UTUMQK6FAYJO5KG6U25OPIXYDXG6J6T4DYX7X7CRUG3Z7ATA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.322713680 (581h14m21.193796725s) [noise=true , address=77.90.31.17:4800 , piece_id=CTFFZF4I7CJ6GOV2BBQHU6LNKKN43CUTYC5EADK5V2CDRWLBHKOA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.322893873 (581h14m21.193976909s) [noise=true , address=145.79.227.33:4885 , piece_id=KKIIGYHXILNGTRLL667N3WYJWJBKFB6DMKD6XOPQ3H6MLJWQX4WA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.323312190 (581h14m21.194395295s) [noise=true , address=77.90.31.11:4580 , piece_id=LPEOROV7YMHMKHC4E3EHOY6Y6CELM3EDXTSVY6MNK3AEJ2OCWXRQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.323394136 (581h14m21.194477181s) [noise=true , address=77.90.31.18:4833 , piece_id=T5LUJEOIVHDT7S6YQAPQNJR5XTZQ47KRD6KFAHPRI7Q76QEF22RA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.323848040 (581h14m21.194931085s) [noise=true , address=64.49.13.27:4958 , piece_id=5BUSTEPZWQLFKYL3FK4LJH4RCBRHT5Y7LIB65JKLPIKBR6LNHWMQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.324101412 (581h14m21.195184447s) [noise=true , address=45.91.116.4:4956 , piece_id=QKZ5H6ZLCVOHVZYM36RI2GW2LQD6ZWNOD2ED6UWJWTHYM2CSZ36Q, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.324762251 (581h14m21.195845346s) [noise=true , address=77.90.31.12:4613 , piece_id=GEVKDWFGSMZWV2RDRPS6FP4N4PKAMOQ7SIAVMTUGRFOSHZSQPEBQ, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:21.326068749 (581h14m21.197151794s) [noise=true , address=145.79.227.40:4062 , piece_id=XBW77GTJSH3JR646ASTU26NFMLYJSOLGGTAN5PPA5XLC5P2LWGAA, seg_pos={0 0}, upload=509538]: Uploading piece... Dec 29 20:43:22.608153795 (581h14m22.47923684s) [upload=509538]: Next returned nil segment Dec 29 20:43:22.608212997 (581h14m22.479296052s) [upload=509538]: Waiting for error group managing segments... Dec 29 20:43:23.010425712 (581h14m22.881508747s) [noise=true , address=64.49.13.140:4511 , piece_id=FX3ROLTTGBGY3PHEDFAW3I4ZS4IWWPIWHZJGS6GLFQGMUZJNTRGQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.069323152 (581h14m22.940406197s) [noise=true , address=64.49.13.27:4958 , piece_id=5BUSTEPZWQLFKYL3FK4LJH4RCBRHT5Y7LIB65JKLPIKBR6LNHWMQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.141204885 (581h14m23.01228793s) [noise=true , address=64.49.13.155:4924 , piece_id=QRB3S2MRNJMXCWFHSVS62BT7BFRLCFALZYVUZPRXQBMQVTYHLYIA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.374116666 (581h14m23.245199711s) [noise=true , address=77.90.31.11:4580 , piece_id=LPEOROV7YMHMKHC4E3EHOY6Y6CELM3EDXTSVY6MNK3AEJ2OCWXRQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.397441589 (581h14m23.268524625s) [noise=true , address=77.90.31.18:4833 , piece_id=T5LUJEOIVHDT7S6YQAPQNJR5XTZQ47KRD6KFAHPRI7Q76QEF22RA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.412860279 (581h14m23.283943324s) [noise=true , address=77.90.31.12:4613 , piece_id=GEVKDWFGSMZWV2RDRPS6FP4N4PKAMOQ7SIAVMTUGRFOSHZSQPEBQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.427598423 (581h14m23.298681468s) [noise=true , address=45.91.116.4:4956 , piece_id=QKZ5H6ZLCVOHVZYM36RI2GW2LQD6ZWNOD2ED6UWJWTHYM2CSZ36Q, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.449350071 (581h14m23.320433106s) [noise=true , address=77.90.31.21:4883 , piece_id=GAMV3JW2MMG4NNVXIIWWKLSNLBJM332XYZILI3CNKHS5JPVTYCSQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:23.739329737 (581h14m23.610412782s) [noise=true , address=77.90.31.17:4800 , piece_id=CTFFZF4I7CJ6GOV2BBQHU6LNKKN43CUTYC5EADK5V2CDRWLBHKOA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.072352183 (581h14m23.943435228s) [noise=true , address=145.79.227.40:4062 , piece_id=XBW77GTJSH3JR646ASTU26NFMLYJSOLGGTAN5PPA5XLC5P2LWGAA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.103723028 (581h14m23.974806063s) [noise=true , address=145.79.227.35:4493 , piece_id=QPUTA52LHETXXVXWOZHXH4UM7JVWHUB63LGQSAI76GTOYFITNNSQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.152376661 (581h14m24.023459696s) [noise=true , address=145.79.227.40:4073 , piece_id=QU5OVXFRPC5QJQEWDO5IDDZPTB4BM2YRYBV2WBZXHRWU5RO7PPFA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.171081338 (581h14m24.042164373s) [noise=true , address=145.79.227.34:4938 , piece_id=6ETRVJX5B2SIHOHNREBMA3MSXG42JPCZ6CF73YW7OULR2J4KWMUQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.178625824 (581h14m24.049708859s) [noise=true , address=145.79.227.43:4187 , piece_id=RZTWWANN4DY26MQBVOXBTSL4KVZPHUZAMDPXZ2FSBSEK7DKHT4FA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: <nil> Dec 29 20:43:24.178650592 (581h14m24.049733627s) [seg_pos={0 0}, upload=509538]: Segment reached optimal threshold of 14 pieces. Dec 29 20:43:24.178999997 (581h14m24.050083042s) [noise=true , address=145.79.227.49:4349 , piece_id=YYPPZJOS44DPDOLTMOLEBO5U6KCYPIJ336K6PPJFLEYTAXALYFLQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: expected piece hash; context canceled; EOF Dec 29 20:43:24.179042908 (581h14m24.050125944s) [noise=true , address=145.79.227.49:4349 , piece_id=YYPPZJOS44DPDOLTMOLEBO5U6KCYPIJ336K6PPJFLEYTAXALYFLQ, seg_pos={0 0}, upload=509538]: Long tail cancellation due to optimal threshold Dec 29 20:43:24.179075631 (581h14m24.050158676s) [noise=true , address=145.79.227.37:4593 , piece_id=5YHXRO4AANIHGG4NTCQG32X36EO3BVNMDOGUV7SUPCPK25QPIVEQ, seg_pos={0 0}, upload=509538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1x5LG8zU8YHJWXdU5ivVsVo7B3esnr816FxoHXDee1jGfFJsMS): protocol: write tcp 188.166.225.170:40880->145.79.227.37:4593: use of closed network connection; write tcp 188.166.225.170:40880->145.79.227.37:4593: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40880->145.79.227.37:4593: use of closed network connection Dec 29 20:43:24.179084057 (581h14m24.050167102s) [noise=true , address=145.79.227.37:4593 , piece_id=5YHXRO4AANIHGG4NTCQG32X36EO3BVNMDOGUV7SUPCPK25QPIVEQ, seg_pos={0 0}, upload=509538]: Long tail cancellation due to optimal threshold Dec 29 20:43:24.179232028 (581h14m24.050315063s) [noise=true , address=145.79.227.7:4322 , piece_id=3ZP5UTUMQK6FAYJO5KG6U25OPIXYDXG6J6T4DYX7X7CRUG3Z7ATA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: write tcp 188.166.225.170:35654->145.79.227.7:4322: use of closed network connection; write tcp 188.166.225.170:35654->145.79.227.7:4322: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35654->145.79.227.7:4322: use of closed network connection Dec 29 20:43:24.179237970 (581h14m24.050321005s) [noise=true , address=145.79.227.7:4322 , piece_id=3ZP5UTUMQK6FAYJO5KG6U25OPIXYDXG6J6T4DYX7X7CRUG3Z7ATA, seg_pos={0 0}, upload=509538]: Long tail cancellation due to optimal threshold Dec 29 20:43:24.179274370 (581h14m24.050357415s) [noise=true , address=145.79.227.33:4885 , piece_id=KKIIGYHXILNGTRLL667N3WYJWJBKFB6DMKD6XOPQ3H6MLJWQX4WA, seg_pos={0 0}, upload=509538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF Dec 29 20:43:24.179285601 (581h14m24.050368636s) [noise=true , address=145.79.227.33:4885 , piece_id=KKIIGYHXILNGTRLL667N3WYJWJBKFB6DMKD6XOPQ3H6MLJWQX4WA, seg_pos={0 0}, upload=509538]: Long tail cancellation due to optimal threshold Dec 29 20:43:24.179309537 (581h14m24.050392572s) [seg_pos={0 0}, upload=509538]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 20:43:24.179417442 (581h14m24.050500477s) [upload=509538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:43:24.179424094 (581h14m24.050507129s) [upload=509538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:43:24.486285315 (581h14m24.35736837s) [upload=509538]: Done uploading segments... Dec 29 20:43:24.486309951 (581h14m24.357392996s) [upload=509538]: Upload finished. err: <nil> Duration: 26.105830023s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)