Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X..................X..XX.........XX..] duration: 5.818584644s FAIL (2025-07-18 15:05:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ad=484544]: Uploading segments... Jul 18 15:03:54.061520924 (605h53m28.832162259s) [upload=484544]: Got next segment. Inline: false Jul 18 15:03:54.061548216 (605h53m28.832189551s) [upload=484544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 15:03:54.061552564 (605h53m28.832193899s) [upload=484544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 15:03:54.292941631 (605h53m29.063582976s) [seg_pos={0 0}, upload=484544]: Begin upload segment... Jul 18 15:03:54.292990295 (605h53m29.06363164s) [seg_pos={0 0}, upload=484544]: Done begin upload segment. Jul 18 15:03:54.293011996 (605h53m29.063653332s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293049207 (605h53m29.063690572s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293387635 (605h53m29.06402897s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293437370 (605h53m29.064078715s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293702057 (605h53m29.064343392s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293785025 (605h53m29.06442637s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294084118 (605h53m29.064725463s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294202274 (605h53m29.064843609s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294384032 (605h53m29.065025367s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294492389 (605h53m29.065133724s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294632507 (605h53m29.065273842s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294751715 (605h53m29.06539305s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294879079 (605h53m29.065520414s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295025358 (605h53m29.065666693s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295118277 (605h53m29.065759612s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295289715 (605h53m29.06593105s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:56.019988911 (605h53m30.790630246s) [upload=484544]: Next returned nil segment Jul 18 15:03:56.020018197 (605h53m30.790659532s) [upload=484544]: Waiting for error group managing segments... Jul 18 15:03:56.340383016 (605h53m31.111024341s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.352400770 (605h53m31.123042105s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.416521375 (605h53m31.1871627s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.427415369 (605h53m31.198056704s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.444465382 (605h53m31.215106727s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.447968033 (605h53m31.218609418s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.459375370 (605h53m31.230016705s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.460150432 (605h53m31.230791777s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.518249386 (605h53m31.288890731s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.521859954 (605h53m31.292501289s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.560237664 (605h53m31.330878989s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:04:23.413509584 (605h53m58.184150909s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection Jul 18 15:04:23.413562105 (605h53m58.18420344s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection Jul 18 15:04:23.413568497 (605h53m58.184209832s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection Jul 18 15:04:23.413609425 (605h53m58.18425076s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection Jul 18 15:04:23.413613753 (605h53m58.184255089s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection Jul 18 15:04:23.413635896 (605h53m58.184277231s) [seg_pos={0 0}, upload=484544]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 15:04:23.657033780 (605h53m58.427675115s) [upload=484544]: Done uploading segments... Jul 18 15:04:23.657061914 (605h53m58.427703249s) [upload=484544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.818613629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X..................X..XX.........XX.] duration: 5.917271256s FAIL (2025-07-18 15:05:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ad=484544]: Uploading segments... Jul 18 15:03:54.061520924 (605h53m28.832162259s) [upload=484544]: Got next segment. Inline: false Jul 18 15:03:54.061548216 (605h53m28.832189551s) [upload=484544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 15:03:54.061552564 (605h53m28.832193899s) [upload=484544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 15:03:54.292941631 (605h53m29.063582976s) [seg_pos={0 0}, upload=484544]: Begin upload segment... Jul 18 15:03:54.292990295 (605h53m29.06363164s) [seg_pos={0 0}, upload=484544]: Done begin upload segment. Jul 18 15:03:54.293011996 (605h53m29.063653332s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293049207 (605h53m29.063690572s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293387635 (605h53m29.06402897s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293437370 (605h53m29.064078715s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293702057 (605h53m29.064343392s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293785025 (605h53m29.06442637s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294084118 (605h53m29.064725463s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294202274 (605h53m29.064843609s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294384032 (605h53m29.065025367s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294492389 (605h53m29.065133724s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294632507 (605h53m29.065273842s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294751715 (605h53m29.06539305s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294879079 (605h53m29.065520414s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295025358 (605h53m29.065666693s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295118277 (605h53m29.065759612s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295289715 (605h53m29.06593105s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:56.019988911 (605h53m30.790630246s) [upload=484544]: Next returned nil segment Jul 18 15:03:56.020018197 (605h53m30.790659532s) [upload=484544]: Waiting for error group managing segments... Jul 18 15:03:56.340383016 (605h53m31.111024341s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.352400770 (605h53m31.123042105s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.416521375 (605h53m31.1871627s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.427415369 (605h53m31.198056704s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.444465382 (605h53m31.215106727s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.447968033 (605h53m31.218609418s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.459375370 (605h53m31.230016705s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.460150432 (605h53m31.230791777s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.518249386 (605h53m31.288890731s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.521859954 (605h53m31.292501289s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.560237664 (605h53m31.330878989s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:04:23.413509584 (605h53m58.184150909s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection Jul 18 15:04:23.413562105 (605h53m58.18420344s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection Jul 18 15:04:23.413568497 (605h53m58.184209832s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection Jul 18 15:04:23.413609425 (605h53m58.18425076s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection Jul 18 15:04:23.413613753 (605h53m58.184255089s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection Jul 18 15:04:23.413635896 (605h53m58.184277231s) [seg_pos={0 0}, upload=484544]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 15:04:23.657033780 (605h53m58.427675115s) [upload=484544]: Done uploading segments... Jul 18 15:04:23.657061914 (605h53m58.427703249s) [upload=484544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.917300873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..................X..XX.........XX] duration: 30.245110552s FAIL (2025-07-18 15:04:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ad=484544]: Uploading segments... Jul 18 15:03:54.061520924 (605h53m28.832162259s) [upload=484544]: Got next segment. Inline: false Jul 18 15:03:54.061548216 (605h53m28.832189551s) [upload=484544]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 15:03:54.061552564 (605h53m28.832193899s) [upload=484544]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 15:03:54.292941631 (605h53m29.063582976s) [seg_pos={0 0}, upload=484544]: Begin upload segment... Jul 18 15:03:54.292990295 (605h53m29.06363164s) [seg_pos={0 0}, upload=484544]: Done begin upload segment. Jul 18 15:03:54.293011996 (605h53m29.063653332s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293049207 (605h53m29.063690572s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293387635 (605h53m29.06402897s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293437370 (605h53m29.064078715s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293702057 (605h53m29.064343392s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.293785025 (605h53m29.06442637s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294084118 (605h53m29.064725463s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294202274 (605h53m29.064843609s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294384032 (605h53m29.065025367s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294492389 (605h53m29.065133724s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294632507 (605h53m29.065273842s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294751715 (605h53m29.06539305s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.294879079 (605h53m29.065520414s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295025358 (605h53m29.065666693s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295118277 (605h53m29.065759612s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:54.295289715 (605h53m29.06593105s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Uploading piece... Jul 18 15:03:56.019988911 (605h53m30.790630246s) [upload=484544]: Next returned nil segment Jul 18 15:03:56.020018197 (605h53m30.790659532s) [upload=484544]: Waiting for error group managing segments... Jul 18 15:03:56.340383016 (605h53m31.111024341s) [noise=true , address=104.36.82.6:4875 , piece_id=YU6Q7SUVFC5MI5BYP4B65RHUEFMRS5EO43MFX7SKWTYNUOTG4MSA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.352400770 (605h53m31.123042105s) [noise=true , address=104.36.82.12:4149 , piece_id=M43XCWKHYLBJEWSMXDGFTR776XGDNGOGAHEIN7QWEYPLJBQFFDRA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.416521375 (605h53m31.1871627s) [noise=true , address=45.91.116.5:4297 , piece_id=REYPQFVSNMXM5RIM2ZB73NE4RWNVBSEE35JCI47EXAZ34VA5NL3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.427415369 (605h53m31.198056704s) [noise=true , address=45.91.116.5:4298 , piece_id=W3ILRNUB42RXOWSAOU7DDZDT2JLVTDRMRDMT3CM2WVTW4KR2KQNA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.444465382 (605h53m31.215106727s) [noise=true , address=104.36.82.2:4732 , piece_id=YMV5JMKM2CYVJ5DSLQW5S5YUUJCEGCQWOJTU5JL366OHRBI2U4PA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.447968033 (605h53m31.218609418s) [noise=true , address=45.91.116.4:4939 , piece_id=4WHS3WOFIIHOHO6VJWU2KKSVDLLZ6RAR3ODEPTZNBTD7CVNLHR3A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.459375370 (605h53m31.230016705s) [noise=true , address=45.91.116.7:4009 , piece_id=3WNQLEIVJXQK4GYHX4OPLQ7SN3ERMQUVFD4BPMNSG3E2RYAUJLQQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.460150432 (605h53m31.230791777s) [noise=true , address=45.91.116.14:4190 , piece_id=5WB5K224JSMXVJFCFEMTJW4WS4ASUK7K5GM2KTMXIW4SPHCUGJ2Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.518249386 (605h53m31.288890731s) [noise=true , address=45.91.116.5:4299 , piece_id=GQJ7SHNIVK6W7X5NT6W6GING7MAV7MMOA4MUCK7WYCNFT3UYA5VQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.521859954 (605h53m31.292501289s) [noise=true , address=45.91.116.9:4262 , piece_id=NOB6J56MLOZH5WRZ2IUMSUROTJ7S4323R4Y5HB5IHX7C7ZV3UCMQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:03:56.560237664 (605h53m31.330878989s) [noise=true , address=45.91.116.10:4083 , piece_id=BR423UB6MSPNE6B6CH6NLONJNY5ANNVSGG3SROBMLA4VVK5MX74Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: <nil> Jul 18 15:04:23.413509584 (605h53m58.184150909s) [noise=true , address=77.90.31.21:4878 , piece_id=2YENPJWRYVFBX66KZFDFVQKAIEXIFWOYGTDS2KMNKPP62T7YBP6Q, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58058->77.90.31.21:4878: use of closed network connection Jul 18 15:04:23.413562105 (605h53m58.18420344s) [noise=true , address=77.90.31.11:4554 , piece_id=LNE5CUFM2YDVE4GTL6GWVR3OWRDQTOR7KFNZJMWCR4ZV3HQQREFQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56390->77.90.31.11:4554: use of closed network connection Jul 18 15:04:23.413568497 (605h53m58.184209832s) [noise=true , address=77.90.31.13:4628 , piece_id=K6DSMTTEKT2QV7BNIU6X5B4ZDPRIU3WBENXH25HDB24XUGNQ5I4A, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54362->77.90.31.13:4628: use of closed network connection Jul 18 15:04:23.413609425 (605h53m58.18425076s) [noise=true , address=77.90.31.17:4770 , piece_id=KWEQQEHKRDQY4Y4XRN6XSAJAS6EQYBWGYAEH7J26PX3AV3KZVVLA, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->77.90.31.17:4770: use of closed network connection Jul 18 15:04:23.413613753 (605h53m58.184255089s) [noise=true , address=77.90.31.18:4805 , piece_id=7OGKC4DOOXSGVSRVBVURSZOKNOKJHAX25QHAPHE5WSMZQS45JCLQ, seg_pos={0 0}, upload=484544]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57114->77.90.31.18:4805: use of closed network connection Jul 18 15:04:23.413635896 (605h53m58.184277231s) [seg_pos={0 0}, upload=484544]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 15:04:23.657033780 (605h53m58.427675115s) [upload=484544]: Done uploading segments... Jul 18 15:04:23.657061914 (605h53m58.427703249s) [upload=484544]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.245125079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..................X..XX.........X] duration: 30.249727187s FAIL (2025-07-18 15:03:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 15:03:04.814436560 (605h52m39.585077895s) [upload=484532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 15:03:05.106110341 (605h52m39.876751677s) [upload=484532]: Done uploading segments... Jul 18 15:03:05.106128366 (605h52m39.876769701s) [upload=484532]: Upload finished. err: <nil> Jul 18 15:03:23.587827852 (605h52m58.358469187s) [upload=484537]: Starting upload Jul 18 15:03:23.587892326 (605h52m58.358533661s) [upload=484537]: Done starting upload Jul 18 15:03:23.587975576 (605h52m58.358616911s) [upload=484537]: Uploading segments... Jul 18 15:03:23.588118659 (605h52m58.358759994s) [upload=484537]: Got next segment. Inline: false Jul 18 15:03:23.588152594 (605h52m58.358793929s) [upload=484537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 15:03:23.588157703 (605h52m58.358799038s) [upload=484537]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 15:03:23.822756592 (605h52m58.593397927s) [seg_pos={0 0}, upload=484537]: Begin upload segment... Jul 18 15:03:23.822800205 (605h52m58.59344154s) [seg_pos={0 0}, upload=484537]: Done begin upload segment. Jul 18 15:03:23.822825113 (605h52m58.593466448s) [noise=true , address=104.36.82.13:4185 , piece_id=ZUN4MZXRVEN5UAOZHRPAZGXZIQZSKVHWP6BWJORJVRW72MSNFPHA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.822869158 (605h52m58.593510493s) [noise=true , address=104.36.82.11:4114 , piece_id=MOMFVY2IXLUG65FVRI76QG4PQZJKEK7NIBLS23GC62ZJ3UWSLYRA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.823191525 (605h52m58.59383286s) [noise=true , address=104.36.82.15:4257 , piece_id=G764YUQV2KAYTI6U2VONYDUJ5MXGM2K5RUT35FEKHROW3HX22OVQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.823525874 (605h52m58.594167209s) [noise=true , address=45.91.116.4:4937 , piece_id=QZPBXEZRC4ILHG5AZOLQN4WBX7RNOZKFNYDSOVA5YN4BPI5Z3MEA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.823539590 (605h52m58.594180925s) [noise=true , address=77.90.31.18:4805 , piece_id=3FZKZ4VAM27QCCBTRLPSCSFYKRKPFNJVJA3JHVNDBK53EI7IOYQQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.823798767 (605h52m58.594440102s) [noise=true , address=77.90.31.16:4735 , piece_id=ZUDF5WRQB7ODJKA3REU3BBNMAXBDB2CZZEWK2PH5FW2NL5AISDHQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.823942601 (605h52m58.594583937s) [noise=true , address=77.90.31.13:4628 , piece_id=E722DADFYX3ZVZQCYIFNW6RPJJF4XL2GZMFTWLLW6WOBPNUJ2KPA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.824095555 (605h52m58.59473689s) [noise=true , address=104.36.82.5:4839 , piece_id=OARLJ4Z5OQKK4YPWIZDSRFFCPV7DUINWHCK5UKJHXKM6PGCGQKRQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.824315045 (605h52m58.59495637s) [noise=true , address=77.90.31.17:4772 , piece_id=O2V5A4NHEQGFC6U2H62K5WGS3VB3A3CE6SWJL46GKFIY56SAUNGQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.824390588 (605h52m58.595031923s) [noise=true , address=104.36.82.16:4295 , piece_id=GFYPOLISBOXGEY6Y3CHTJ7K4V6VOQFROVRLVUB7JBL4SWBEC43YA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.824673460 (605h52m58.595314785s) [noise=true , address=45.91.116.11:4117 , piece_id=KDBMS22ZL3EAAHQ2PAKL2QJXKSEJZUMHIXH2DCX7XOGM6XRPQAWA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.825309217 (605h52m58.595950552s) [noise=true , address=77.90.31.15:4697 , piece_id=RAXERBMO3D2BVYMR2UHEPS3ZC5T7KUYSC7I7DCAYJ2BH7357VFVQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.826038331 (605h52m58.596679666s) [noise=true , address=77.90.31.12:4590 , piece_id=BTKQVIOKZ2VIXEVIYIKSK7DOUZIWJC7B4DPAK35VW6PVFKJSKK5A, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.826089790 (605h52m58.596731125s) [noise=true , address=45.91.116.12:4226 , piece_id=KEOEFMLF2PBHITYQ5HM4DPRMMKC4FF4X2UO2WMSM7SDTF446DWDA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.826448927 (605h52m58.597090262s) [noise=true , address=104.36.82.10:4077 , piece_id=Q4VYLH7NTWADMC5G6AKSY2LTOQOFCE2MDPMCIOO64REV35PVMVIQ, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:23.826831018 (605h52m58.597472353s) [noise=true , address=104.36.82.4:4804 , piece_id=EGWG224DJXKFWQEQFA4OBRGHC23EY5R4ZCDSQ46PN44QSXDDPKNA, seg_pos={0 0}, upload=484537]: Uploading piece... Jul 18 15:03:25.543974546 (605h53m0.314615881s) [upload=484537]: Next returned nil segment Jul 18 15:03:25.543991659 (605h53m0.314632994s) [upload=484537]: Waiting for error group managing segments... Jul 18 15:03:25.866260821 (605h53m0.636902156s) [noise=true , address=104.36.82.16:4295 , piece_id=GFYPOLISBOXGEY6Y3CHTJ7K4V6VOQFROVRLVUB7JBL4SWBEC43YA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.870918043 (605h53m0.641559378s) [noise=true , address=45.91.116.12:4226 , piece_id=KEOEFMLF2PBHITYQ5HM4DPRMMKC4FF4X2UO2WMSM7SDTF446DWDA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.876868388 (605h53m0.647509723s) [noise=true , address=45.91.116.11:4117 , piece_id=KDBMS22ZL3EAAHQ2PAKL2QJXKSEJZUMHIXH2DCX7XOGM6XRPQAWA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.884973175 (605h53m0.65561451s) [noise=true , address=104.36.82.5:4839 , piece_id=OARLJ4Z5OQKK4YPWIZDSRFFCPV7DUINWHCK5UKJHXKM6PGCGQKRQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.902289817 (605h53m0.672931142s) [noise=true , address=45.91.116.4:4937 , piece_id=QZPBXEZRC4ILHG5AZOLQN4WBX7RNOZKFNYDSOVA5YN4BPI5Z3MEA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.945042188 (605h53m0.715683513s) [noise=true , address=104.36.82.13:4185 , piece_id=ZUN4MZXRVEN5UAOZHRPAZGXZIQZSKVHWP6BWJORJVRW72MSNFPHA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.977226237 (605h53m0.747867572s) [noise=true , address=104.36.82.10:4077 , piece_id=Q4VYLH7NTWADMC5G6AKSY2LTOQOFCE2MDPMCIOO64REV35PVMVIQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.979292390 (605h53m0.749933725s) [noise=true , address=104.36.82.15:4257 , piece_id=G764YUQV2KAYTI6U2VONYDUJ5MXGM2K5RUT35FEKHROW3HX22OVQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:25.982292371 (605h53m0.752933706s) [noise=true , address=104.36.82.11:4114 , piece_id=MOMFVY2IXLUG65FVRI76QG4PQZJKEK7NIBLS23GC62ZJ3UWSLYRA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:26.093439904 (605h53m0.864081239s) [noise=true , address=104.36.82.4:4804 , piece_id=EGWG224DJXKFWQEQFA4OBRGHC23EY5R4ZCDSQ46PN44QSXDDPKNA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:42.243212778 (605h53m17.013854113s) [noise=true , address=77.90.31.12:4590 , piece_id=BTKQVIOKZ2VIXEVIYIKSK7DOUZIWJC7B4DPAK35VW6PVFKJSKK5A, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:43.895708066 (605h53m18.666349391s) [noise=true , address=77.90.31.17:4772 , piece_id=O2V5A4NHEQGFC6U2H62K5WGS3VB3A3CE6SWJL46GKFIY56SAUNGQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: <nil> Jul 18 15:03:52.940060522 (605h53m27.710701927s) [noise=true , address=77.90.31.18:4805 , piece_id=3FZKZ4VAM27QCCBTRLPSCSFYKRKPFNJVJA3JHVNDBK53EI7IOYQQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:35636->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:35636->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35636->77.90.31.18:4805: use of closed network connection Jul 18 15:03:52.940150224 (605h53m27.710791559s) [noise=true , address=77.90.31.16:4735 , piece_id=ZUDF5WRQB7ODJKA3REU3BBNMAXBDB2CZZEWK2PH5FW2NL5AISDHQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:58630->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58630->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58630->77.90.31.16:4735: use of closed network connection Jul 18 15:03:52.940322123 (605h53m27.710963508s) [noise=true , address=77.90.31.15:4697 , piece_id=RAXERBMO3D2BVYMR2UHEPS3ZC5T7KUYSC7I7DCAYJ2BH7357VFVQ, seg_pos={0 0}, upload=484537]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:49562->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:49562->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49562->77.90.31.15:4697: use of closed network connection Jul 18 15:03:52.940602469 (605h53m27.711243854s) [noise=true , address=77.90.31.13:4628 , piece_id=E722DADFYX3ZVZQCYIFNW6RPJJF4XL2GZMFTWLLW6WOBPNUJ2KPA, seg_pos={0 0}, upload=484537]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:48614->77.90.31.13:4628: use of closed network connection; context deadline exceeded Jul 18 15:03:52.940644088 (605h53m27.711285473s) [seg_pos={0 0}, upload=484537]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 15:03:53.188738379 (605h53m27.959379714s) [upload=484537]: Done uploading segments... Jul 18 15:03:53.188763707 (605h53m27.959405042s) [upload=484537]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.249750582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X..................X..XX.........] duration: 6.005928407s FAIL (2025-07-18 15:03:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.005958926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X..................X..XX........] duration: 5.836621555s FAIL (2025-07-18 15:02:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.836647723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X..................X..XX.......] duration: 5.777452831s FAIL (2025-07-18 15:02:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.777483008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X..................X..XX......] duration: 6.194157054s FAIL (2025-07-18 15:01:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.194189947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X..................X..XX.....] duration: 6.023299092s FAIL (2025-07-18 15:00:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.023327066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X..................X..XX....] duration: 5.790099624s FAIL (2025-07-18 14:59:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790122788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X..................X..XX...] duration: 5.792850639s FAIL (2025-07-18 14:59:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.792878081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X..................X..XX..] duration: 6.30411279s FAIL (2025-07-18 14:58:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.304140192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X..................X..XX.] duration: 5.979164184s FAIL (2025-07-18 14:58:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.979189271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X..................X..XX] duration: 30.238458488s FAIL (2025-07-18 14:57:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== .135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 18 14:57:28.509758608 (605h47m3.280399943s) [upload=484457]: Starting upload Jul 18 14:57:28.509858639 (605h47m3.280499974s) [upload=484457]: Done starting upload Jul 18 14:57:28.509927962 (605h47m3.280569297s) [upload=484457]: Uploading segments... Jul 18 14:57:28.510133395 (605h47m3.28077473s) [upload=484457]: Got next segment. Inline: false Jul 18 14:57:28.510169104 (605h47m3.280810439s) [upload=484457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:57:28.510173993 (605h47m3.280815328s) [upload=484457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:57:28.739206232 (605h47m3.509847557s) [seg_pos={0 0}, upload=484457]: Begin upload segment... Jul 18 14:57:28.739306533 (605h47m3.509947868s) [seg_pos={0 0}, upload=484457]: Done begin upload segment. Jul 18 14:57:28.739333024 (605h47m3.509974369s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739341761 (605h47m3.509983096s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739728641 (605h47m3.510369966s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.739744351 (605h47m3.510385686s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740040287 (605h47m3.510681622s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740103518 (605h47m3.510744853s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740440173 (605h47m3.511081508s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740654323 (605h47m3.511295658s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.740779862 (605h47m3.511421197s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741116667 (605h47m3.511758002s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741126276 (605h47m3.511767621s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.741588841 (605h47m3.512230176s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.744126005 (605h47m3.51476734s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.746619606 (605h47m3.517260941s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747092591 (605h47m3.517733926s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:28.747796919 (605h47m3.518438254s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Uploading piece... Jul 18 14:57:30.466367525 (605h47m5.23700886s) [upload=484457]: Next returned nil segment Jul 18 14:57:30.466386101 (605h47m5.237027436s) [upload=484457]: Waiting for error group managing segments... Jul 18 14:57:30.779197037 (605h47m5.549838372s) [noise=true , address=45.91.116.14:4191 , piece_id=XDOL3LRGDCESKVAIVE2RUJDQFUPSKYL42VXVB646YWVGZXEOAIOQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792104764 (605h47m5.562746099s) [noise=true , address=45.91.116.9:4263 , piece_id=QQA4MA2DF2UY3DAZSUG2SWOKDBUHFFLPTVYX4XPTLMGJ44HVIK5A, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.792694021 (605h47m5.563335356s) [noise=true , address=45.91.116.10:4081 , piece_id=SJXGREHLTR4DE4S3JUIDKJWNGYPYAHKSW567EZ2G24XJYHXCEBDQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.793282928 (605h47m5.563924264s) [noise=true , address=45.91.116.8:4048 , piece_id=LNI2ONKK5KYKK5BWFECLY7RDJ3FWGWZLI5A6S2CYF7ZV6JUFY6EA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797169465 (605h47m5.5678108s) [noise=true , address=45.91.116.4:4937 , piece_id=OQ2VZ34E2KOTF4R2HGSB6XS3RJQJI64UMLJPMT43MF6JNJUMSCMA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.797356482 (605h47m5.567997817s) [noise=true , address=45.91.116.8:4045 , piece_id=X7S5DTXD22EU5LX6KX5VZHQRSFLVNM73CHKDLD3UGZ2UZB7RFRIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.808140206 (605h47m5.578781541s) [noise=true , address=45.91.116.12:4227 , piece_id=ZYWQFZ3KJQNQOCECNEKY5OOZKOBVY7UOB3GYAXUSZSKLZJNGLHTQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.830720279 (605h47m5.601361614s) [noise=true , address=45.91.116.6:4974 , piece_id=AGKYMLWYUTN5XGA6DPZTK5FBVCRCPF23X5F3RN5DQMTO626PBFXQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.864852807 (605h47m5.635494142s) [noise=true , address=104.36.82.14:4221 , piece_id=7ICKCII4AG5EFKCYCMQMZVM5AZDNTBT7UNBXNC6VLRWSDHGPSMKA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.874905682 (605h47m5.645547007s) [noise=true , address=104.36.82.16:4295 , piece_id=W55VULTDHBU2KOK6GAVFI4MCFK2MBWTV57GF65L3WXAMGITFTEPA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.893044829 (605h47m5.663686164s) [noise=true , address=104.36.82.6:4874 , piece_id=LPTNCLGJRM3V5A5HUBUAJR76PJNTX7AUDVSEYAMG5HBNDTZZ35UA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:30.968868696 (605h47m5.739510031s) [noise=true , address=104.36.82.9:4006 , piece_id=Z5CH3ICFYCTGEOUDUSTSRJIOI43IRGCDJ2JKY2S2QQJ6JPM6HJWQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: <nil> Jul 18 14:57:57.870784008 (605h47m32.641425363s) [noise=true , address=77.90.31.10:4517 , piece_id=6S6N4ZH3L6DZBX4VDMDMSNLTHIP4ZSX4RU3U3JJTPUBW4MXJ44GA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50056->77.90.31.10:4517: use of closed network connection Jul 18 14:57:57.870844503 (605h47m32.641485838s) [noise=true , address=77.90.31.16:4734 , piece_id=RNDRUWXCJFJ33ZNP3JOU725W33UCVD4AOIHO7CSPGCQK4JK7YZNA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58654->77.90.31.16:4734: use of closed network connection Jul 18 14:57:57.870891964 (605h47m32.641533299s) [noise=true , address=77.90.31.21:4878 , piece_id=ODLL4VSVUCP4VJY5IWZAASKMGJZGRHGNHOPX76O64KZRSC3VI4TQ, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59830->77.90.31.21:4878: use of closed network connection Jul 18 14:57:57.870901863 (605h47m32.641543198s) [noise=true , address=77.90.31.18:4805 , piece_id=HKOWNOO2KHZBOZ2TILTQ33OT7J2TZKKKUX7Y27GJMDWJL3HE6OIA, seg_pos={0 0}, upload=484457]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38988->77.90.31.18:4805: use of closed network connection Jul 18 14:57:57.870948082 (605h47m32.641589417s) [seg_pos={0 0}, upload=484457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:57:58.107929911 (605h47m32.878571246s) [upload=484457]: Done uploading segments... Jul 18 14:57:58.107956943 (605h47m32.878598278s) [upload=484457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.238485589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X..................X..X] duration: 30.242903621s FAIL (2025-07-18 14:56:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 5791 (605h45m41.403997136s) [upload=484440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:56:06.633363305 (605h45m41.404004651s) [upload=484440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:56:06.921138952 (605h45m41.691780277s) [upload=484440]: Done uploading segments... Jul 18 14:56:06.921157937 (605h45m41.691799262s) [upload=484440]: Upload finished. err: <nil> Jul 18 14:56:22.772512942 (605h45m57.543154277s) [upload=484445]: Starting upload Jul 18 14:56:22.772623604 (605h45m57.543264959s) [upload=484445]: Done starting upload Jul 18 14:56:22.772718375 (605h45m57.54335971s) [upload=484445]: Uploading segments... Jul 18 14:56:22.773044920 (605h45m57.543686255s) [upload=484445]: Got next segment. Inline: false Jul 18 14:56:22.773078354 (605h45m57.543719739s) [upload=484445]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:56:22.773086539 (605h45m57.543727874s) [upload=484445]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:56:23.015792070 (605h45m57.786433405s) [seg_pos={0 0}, upload=484445]: Begin upload segment... Jul 18 14:56:23.015839601 (605h45m57.786480936s) [seg_pos={0 0}, upload=484445]: Done begin upload segment. Jul 18 14:56:23.015860421 (605h45m57.786501756s) [noise=true , address=45.91.116.5:4299 , piece_id=AVADNC4KEFF72J6GMJPY7VPXA4KMDNXZ4ZKXDS5JVAL756RY6JHQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.015911529 (605h45m57.786552864s) [noise=true , address=45.91.116.13:4153 , piece_id=X7YZYRTVVSFIGC3FH3IULTXCPHJGEVB4YN2H3VACSE5J5T2S7YUA, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.016205852 (605h45m57.786847187s) [noise=true , address=77.90.31.21:4878 , piece_id=XTNFQHBCGE3TWBR6GJF24NKH25EZV4BGKBFU7QIMO2KAHIICCIPA, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.016596660 (605h45m57.787237995s) [noise=true , address=77.90.31.13:4626 , piece_id=4336C4HFSE4DEXKZGVPU6CMTK3Q356MU5KEEXHBOWLKXUYD2SJEA, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.016777555 (605h45m57.78741889s) [noise=true , address=45.91.116.11:4118 , piece_id=WRP7G4PKS3D6RODEF3FDIYVCZZ72LWATQVYD4AGA6FNRF3BLFB3Q, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.016877727 (605h45m57.787519062s) [noise=true , address=77.90.31.17:4770 , piece_id=MXZ2HT2NLOEFDUP62HW6N2EAYM47R37KMD4NXWOVP6AU6M2YQDAQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.017181188 (605h45m57.787822523s) [noise=true , address=77.90.31.18:4805 , piece_id=GMJBVBGAEHWPNICH2KFBM6YPD2WW7EZI6X35GALP6MHGQOA3D7JQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.017398484 (605h45m57.788039819s) [noise=true , address=45.91.116.4:4937 , piece_id=ICCJNJHADRDK46RMHOXJMXSKCAXHBPHI5NRBUCP6RZTCBXDTYTAQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.017605319 (605h45m57.788246654s) [noise=true , address=45.91.116.13:4155 , piece_id=64HMWSCXCZOY42VXLY4ERU7PY5RLXICGLH7RSR3ZAREYNNCYESQQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.018047164 (605h45m57.788688509s) [noise=true , address=104.36.82.8:4971 , piece_id=2PQAX7Y64FGW6CAM5BSNBZXMS37547WXVYDOC6LHVB6CFMK5YW3A, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.018405861 (605h45m57.789047207s) [noise=true , address=45.91.116.14:4190 , piece_id=SF7VRLU3NY3RWY6WMCHKTZIUBOUVGXH4LUY62PDWSYNQRBBJ6QNQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.018721575 (605h45m57.78936291s) [noise=true , address=45.91.116.10:4082 , piece_id=PIRXBDSZFL3KHWXWZARL46TP6S3WMSFXPTCWX4NPXAZQEQQ7NC7Q, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.018740852 (605h45m57.789382187s) [noise=true , address=77.90.31.10:4518 , piece_id=HQGQ32ROEQTTS6EQTHWC2GM5YIYNV47FWAHCXP5OVFHDTY4VTIIQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.019102845 (605h45m57.78974418s) [noise=true , address=104.36.82.5:4837 , piece_id=LHCKRC7M34QW6Z7IG73BYCCROGW26KFQL4MU73B4OWGPFKQN477Q, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.019422366 (605h45m57.790063701s) [noise=true , address=104.36.82.8:4972 , piece_id=4IT3HDPVDGSY4YEHVRXFU2HUOOKGEP7IJMNR4D2I37XGHOLVJJPQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:23.019680600 (605h45m57.790321935s) [noise=true , address=45.91.116.9:4263 , piece_id=F2JBB4YFZDTKLH7N7OGHVMDPIHRL5OD3C7HIHIK2ACLATPLQ42QQ, seg_pos={0 0}, upload=484445]: Uploading piece... Jul 18 14:56:24.762190462 (605h45m59.532831797s) [upload=484445]: Next returned nil segment Jul 18 14:56:24.762211102 (605h45m59.532852437s) [upload=484445]: Waiting for error group managing segments... Jul 18 14:56:25.062933880 (605h45m59.833575205s) [noise=true , address=104.36.82.5:4837 , piece_id=LHCKRC7M34QW6Z7IG73BYCCROGW26KFQL4MU73B4OWGPFKQN477Q, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.065399728 (605h45m59.836041063s) [noise=true , address=45.91.116.14:4190 , piece_id=SF7VRLU3NY3RWY6WMCHKTZIUBOUVGXH4LUY62PDWSYNQRBBJ6QNQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.076047480 (605h45m59.846688825s) [noise=true , address=45.91.116.10:4082 , piece_id=PIRXBDSZFL3KHWXWZARL46TP6S3WMSFXPTCWX4NPXAZQEQQ7NC7Q, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.083122197 (605h45m59.853763522s) [noise=true , address=45.91.116.5:4299 , piece_id=AVADNC4KEFF72J6GMJPY7VPXA4KMDNXZ4ZKXDS5JVAL756RY6JHQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.083268537 (605h45m59.853909872s) [noise=true , address=45.91.116.9:4263 , piece_id=F2JBB4YFZDTKLH7N7OGHVMDPIHRL5OD3C7HIHIK2ACLATPLQ42QQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.088631018 (605h45m59.859272363s) [noise=true , address=45.91.116.11:4118 , piece_id=WRP7G4PKS3D6RODEF3FDIYVCZZ72LWATQVYD4AGA6FNRF3BLFB3Q, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.152388218 (605h45m59.923029543s) [noise=true , address=45.91.116.13:4153 , piece_id=X7YZYRTVVSFIGC3FH3IULTXCPHJGEVB4YN2H3VACSE5J5T2S7YUA, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.161081039 (605h45m59.931722374s) [noise=true , address=45.91.116.4:4937 , piece_id=ICCJNJHADRDK46RMHOXJMXSKCAXHBPHI5NRBUCP6RZTCBXDTYTAQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.167866993 (605h45m59.938508328s) [noise=true , address=45.91.116.13:4155 , piece_id=64HMWSCXCZOY42VXLY4ERU7PY5RLXICGLH7RSR3ZAREYNNCYESQQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.238940632 (605h46m0.009581968s) [noise=true , address=104.36.82.8:4971 , piece_id=2PQAX7Y64FGW6CAM5BSNBZXMS37547WXVYDOC6LHVB6CFMK5YW3A, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:25.486476726 (605h46m0.257118061s) [noise=true , address=104.36.82.8:4972 , piece_id=4IT3HDPVDGSY4YEHVRXFU2HUOOKGEP7IJMNR4D2I37XGHOLVJJPQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:41.518193020 (605h46m16.288834345s) [noise=true , address=77.90.31.21:4878 , piece_id=XTNFQHBCGE3TWBR6GJF24NKH25EZV4BGKBFU7QIMO2KAHIICCIPA, seg_pos={0 0}, upload=484445]: Done uploading piece. err: <nil> Jul 18 14:56:52.123645953 (605h46m26.894287288s) [noise=true , address=77.90.31.18:4805 , piece_id=GMJBVBGAEHWPNICH2KFBM6YPD2WW7EZI6X35GALP6MHGQOA3D7JQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:56:52.123717470 (605h46m26.894358805s) [noise=true , address=77.90.31.13:4626 , piece_id=4336C4HFSE4DEXKZGVPU6CMTK3Q356MU5KEEXHBOWLKXUYD2SJEA, seg_pos={0 0}, upload=484445]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:60144->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:60144->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60144->77.90.31.13:4626: use of closed network connection Jul 18 14:56:52.123799517 (605h46m26.894440862s) [noise=true , address=77.90.31.10:4518 , piece_id=HQGQ32ROEQTTS6EQTHWC2GM5YIYNV47FWAHCXP5OVFHDTY4VTIIQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:42316->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:42316->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42316->77.90.31.10:4518: use of closed network connection Jul 18 14:56:52.123846397 (605h46m26.894487732s) [noise=true , address=77.90.31.17:4770 , piece_id=MXZ2HT2NLOEFDUP62HW6N2EAYM47R37KMD4NXWOVP6AU6M2YQDAQ, seg_pos={0 0}, upload=484445]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:37842->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:37842->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37842->77.90.31.17:4770: use of closed network connection Jul 18 14:56:52.123871074 (605h46m26.894512399s) [seg_pos={0 0}, upload=484445]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:56:52.365194310 (605h46m27.135835645s) [upload=484445]: Done uploading segments... Jul 18 14:56:52.365220200 (605h46m27.135861525s) [upload=484445]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242919081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..................X..] duration: 15.463769957s FAIL (2025-07-18 14:56:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :54:32.673465772 (605h44m7.444107107s) [upload=484418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:54:32.673470350 (605h44m7.444111696s) [upload=484418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:54:33.033377870 (605h44m7.804019205s) [upload=484418]: Done uploading segments... Jul 18 14:54:33.033390935 (605h44m7.80403226s) [upload=484418]: Upload finished. err: <nil> Jul 18 14:54:57.197049716 (605h44m31.967691051s) [upload=484425]: Starting upload Jul 18 14:54:57.197242816 (605h44m31.967884141s) [upload=484425]: Done starting upload Jul 18 14:54:57.197333729 (605h44m31.967975074s) [upload=484425]: Uploading segments... Jul 18 14:54:57.197572877 (605h44m31.968214212s) [upload=484425]: Got next segment. Inline: false Jul 18 14:54:57.197608816 (605h44m31.968250151s) [upload=484425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:54:57.197613995 (605h44m31.96825533s) [upload=484425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:54:57.442567036 (605h44m32.213208422s) [seg_pos={0 0}, upload=484425]: Begin upload segment... Jul 18 14:54:57.442618575 (605h44m32.21325991s) [seg_pos={0 0}, upload=484425]: Done begin upload segment. Jul 18 14:54:57.442643814 (605h44m32.213285149s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.442682567 (605h44m32.213323902s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443085098 (605h44m32.213726433s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443160152 (605h44m32.213801487s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443359022 (605h44m32.214000357s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443682581 (605h44m32.214323906s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443924584 (605h44m32.21456592s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444120259 (605h44m32.214761594s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444516597 (605h44m32.215157942s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444997117 (605h44m32.215638452s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.445888131 (605h44m32.216529466s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446106198 (605h44m32.216747533s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446391223 (605h44m32.217032558s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446600264 (605h44m32.217241589s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446797310 (605h44m32.217438645s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.447108997 (605h44m32.217750352s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:59.198329044 (605h44m33.968970369s) [upload=484425]: Next returned nil segment Jul 18 14:54:59.198346267 (605h44m33.968987602s) [upload=484425]: Waiting for error group managing segments... Jul 18 14:54:59.485963729 (605h44m34.256605055s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498620407 (605h44m34.269261742s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498932524 (605h44m34.269573859s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.499858616 (605h44m34.270499951s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.500862116 (605h44m34.271503451s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.506736285 (605h44m34.27737762s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.565337219 (605h44m34.335978544s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.586833900 (605h44m34.357475235s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.613123812 (605h44m34.383765147s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.652660971 (605h44m34.423302306s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.656263513 (605h44m34.426904849s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.678271101 (605h44m34.448912426s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:55:26.542749239 (605h45m1.313390574s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:55:26.542950234 (605h45m1.313591569s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection Jul 18 14:55:26.543013886 (605h45m1.313655221s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection Jul 18 14:55:26.543061257 (605h45m1.313702592s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection Jul 18 14:55:26.543077878 (605h45m1.313719213s) [seg_pos={0 0}, upload=484425]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:55:26.782046600 (605h45m1.552687945s) [upload=484425]: Done uploading segments... Jul 18 14:55:26.782118086 (605h45m1.552759421s) [upload=484425]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.463786049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..................X.] duration: 25.386970516s FAIL (2025-07-18 14:55:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :54:32.673465772 (605h44m7.444107107s) [upload=484418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:54:32.673470350 (605h44m7.444111696s) [upload=484418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:54:33.033377870 (605h44m7.804019205s) [upload=484418]: Done uploading segments... Jul 18 14:54:33.033390935 (605h44m7.80403226s) [upload=484418]: Upload finished. err: <nil> Jul 18 14:54:57.197049716 (605h44m31.967691051s) [upload=484425]: Starting upload Jul 18 14:54:57.197242816 (605h44m31.967884141s) [upload=484425]: Done starting upload Jul 18 14:54:57.197333729 (605h44m31.967975074s) [upload=484425]: Uploading segments... Jul 18 14:54:57.197572877 (605h44m31.968214212s) [upload=484425]: Got next segment. Inline: false Jul 18 14:54:57.197608816 (605h44m31.968250151s) [upload=484425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:54:57.197613995 (605h44m31.96825533s) [upload=484425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:54:57.442567036 (605h44m32.213208422s) [seg_pos={0 0}, upload=484425]: Begin upload segment... Jul 18 14:54:57.442618575 (605h44m32.21325991s) [seg_pos={0 0}, upload=484425]: Done begin upload segment. Jul 18 14:54:57.442643814 (605h44m32.213285149s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.442682567 (605h44m32.213323902s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443085098 (605h44m32.213726433s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443160152 (605h44m32.213801487s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443359022 (605h44m32.214000357s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443682581 (605h44m32.214323906s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443924584 (605h44m32.21456592s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444120259 (605h44m32.214761594s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444516597 (605h44m32.215157942s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444997117 (605h44m32.215638452s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.445888131 (605h44m32.216529466s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446106198 (605h44m32.216747533s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446391223 (605h44m32.217032558s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446600264 (605h44m32.217241589s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446797310 (605h44m32.217438645s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.447108997 (605h44m32.217750352s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:59.198329044 (605h44m33.968970369s) [upload=484425]: Next returned nil segment Jul 18 14:54:59.198346267 (605h44m33.968987602s) [upload=484425]: Waiting for error group managing segments... Jul 18 14:54:59.485963729 (605h44m34.256605055s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498620407 (605h44m34.269261742s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498932524 (605h44m34.269573859s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.499858616 (605h44m34.270499951s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.500862116 (605h44m34.271503451s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.506736285 (605h44m34.27737762s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.565337219 (605h44m34.335978544s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.586833900 (605h44m34.357475235s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.613123812 (605h44m34.383765147s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.652660971 (605h44m34.423302306s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.656263513 (605h44m34.426904849s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.678271101 (605h44m34.448912426s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:55:26.542749239 (605h45m1.313390574s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:55:26.542950234 (605h45m1.313591569s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection Jul 18 14:55:26.543013886 (605h45m1.313655221s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection Jul 18 14:55:26.543061257 (605h45m1.313702592s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection Jul 18 14:55:26.543077878 (605h45m1.313719213s) [seg_pos={0 0}, upload=484425]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:55:26.782046600 (605h45m1.552687945s) [upload=484425]: Done uploading segments... Jul 18 14:55:26.782118086 (605h45m1.552759421s) [upload=484425]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.386996606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..................X] duration: 30.24122118s FAIL (2025-07-18 14:55:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== :54:32.673465772 (605h44m7.444107107s) [upload=484418]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:54:32.673470350 (605h44m7.444111696s) [upload=484418]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:54:33.033377870 (605h44m7.804019205s) [upload=484418]: Done uploading segments... Jul 18 14:54:33.033390935 (605h44m7.80403226s) [upload=484418]: Upload finished. err: <nil> Jul 18 14:54:57.197049716 (605h44m31.967691051s) [upload=484425]: Starting upload Jul 18 14:54:57.197242816 (605h44m31.967884141s) [upload=484425]: Done starting upload Jul 18 14:54:57.197333729 (605h44m31.967975074s) [upload=484425]: Uploading segments... Jul 18 14:54:57.197572877 (605h44m31.968214212s) [upload=484425]: Got next segment. Inline: false Jul 18 14:54:57.197608816 (605h44m31.968250151s) [upload=484425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:54:57.197613995 (605h44m31.96825533s) [upload=484425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:54:57.442567036 (605h44m32.213208422s) [seg_pos={0 0}, upload=484425]: Begin upload segment... Jul 18 14:54:57.442618575 (605h44m32.21325991s) [seg_pos={0 0}, upload=484425]: Done begin upload segment. Jul 18 14:54:57.442643814 (605h44m32.213285149s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.442682567 (605h44m32.213323902s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443085098 (605h44m32.213726433s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443160152 (605h44m32.213801487s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443359022 (605h44m32.214000357s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443682581 (605h44m32.214323906s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.443924584 (605h44m32.21456592s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444120259 (605h44m32.214761594s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444516597 (605h44m32.215157942s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.444997117 (605h44m32.215638452s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.445888131 (605h44m32.216529466s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446106198 (605h44m32.216747533s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446391223 (605h44m32.217032558s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446600264 (605h44m32.217241589s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.446797310 (605h44m32.217438645s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:57.447108997 (605h44m32.217750352s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Uploading piece... Jul 18 14:54:59.198329044 (605h44m33.968970369s) [upload=484425]: Next returned nil segment Jul 18 14:54:59.198346267 (605h44m33.968987602s) [upload=484425]: Waiting for error group managing segments... Jul 18 14:54:59.485963729 (605h44m34.256605055s) [noise=true , address=104.36.82.2:4732 , piece_id=GERI2BKEMY2TPGR5F6YDNOZR5U4AEF4MHL4D4RXBJ2UCOKRRRB7Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498620407 (605h44m34.269261742s) [noise=true , address=45.91.116.9:4262 , piece_id=Q3XX36SVOYOO7AFOI335DCX33JJPLVBBTXUUUGCW36HRDAE6YJZQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.498932524 (605h44m34.269573859s) [noise=true , address=45.91.116.7:4009 , piece_id=NTOAREUY3SORIGDJBYUWBWB6V4KPYIBWBZWNUWCRDHXXZWAYE2SA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.499858616 (605h44m34.270499951s) [noise=true , address=45.91.116.11:4119 , piece_id=RDEIJMGDNLSIKD4JFLWQWYNXMQKYOWQXM4GDNUTZEEAT3TMNFL2A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.500862116 (605h44m34.271503451s) [noise=true , address=45.91.116.6:4975 , piece_id=ORDZPCETOS4HQUQBJYBKZQHAPPP6FI6C5UNKSCVT3QTGBGV53U3A, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.506736285 (605h44m34.27737762s) [noise=true , address=45.91.116.12:4225 , piece_id=FQDQTDYAZWYRMGG7RD2KDJPCFAADYHDEF4OI2JGUZ63MB6ZPPKEQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.565337219 (605h44m34.335978544s) [noise=true , address=104.36.82.13:4186 , piece_id=XFBAHQLWEIYDN6XK542CNI7GKUAKIATHWKUIKRNOGNX5CTOVPMSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.586833900 (605h44m34.357475235s) [noise=true , address=45.91.116.13:4154 , piece_id=X7CZ3QD42KKLC2IRFFJFPDIFI7RZUNBNUPKIDQPSPR76TQNIJTSA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.613123812 (605h44m34.383765147s) [noise=true , address=45.91.116.4:4938 , piece_id=JQYS2HOJM3Q2ZAP4RTZFRB7ZFPCWP32OG4VAL3A62RLECNMUSFWQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.652660971 (605h44m34.423302306s) [noise=true , address=45.91.116.9:4261 , piece_id=PYUBT37766HA45Q4Q456YAUEE2OYK6G5HG5JJP5U3ASYGYDXJKOA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.656263513 (605h44m34.426904849s) [noise=true , address=104.36.82.6:4875 , piece_id=MOPVV4IXIGIU7OFTZVOD3BOR663G6XCWAM3EAEOZT4BD6WZS2SIQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:54:59.678271101 (605h44m34.448912426s) [noise=true , address=104.36.82.13:4185 , piece_id=A5ZQBN6BOZU6ZE4QJWQI6M5LWWYGHHNY5ALNLBHWRFZEHNUP4ZGA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: <nil> Jul 18 14:55:26.542749239 (605h45m1.313390574s) [noise=true , address=77.90.31.21:4878 , piece_id=AXMTYBJHQVL2ZFNDAY2FOBQANHACZHPUJDPJSYZNGG5RJ73MA24Q, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:55:26.542950234 (605h45m1.313591569s) [noise=true , address=77.90.31.13:4627 , piece_id=2MDSKQTXUWJSYECLI5CEVMRHB36MTCQA2D7PHTCP3K7SEF2DHXVQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59618->77.90.31.13:4627: use of closed network connection Jul 18 14:55:26.543013886 (605h45m1.313655221s) [noise=true , address=77.90.31.19:4843 , piece_id=BPST5JOEEMG5LAZ3A2HBFECVBQBMUVKRPQGGJIT4QQ6V6KMN7ZBA, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48844->77.90.31.19:4843: use of closed network connection Jul 18 14:55:26.543061257 (605h45m1.313702592s) [noise=true , address=77.90.31.12:4589 , piece_id=BZIM7ZJ5JFKUAY5RY6LTCVJ3IS5LFX4UHCIRIVZ4UNJS5NE665MQ, seg_pos={0 0}, upload=484425]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53972->77.90.31.12:4589: use of closed network connection Jul 18 14:55:26.543077878 (605h45m1.313719213s) [seg_pos={0 0}, upload=484425]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:55:26.782046600 (605h45m1.552687945s) [upload=484425]: Done uploading segments... Jul 18 14:55:26.782118086 (605h45m1.552759421s) [upload=484425]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241251098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.850340961s FAIL (2025-07-18 14:54:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.850374366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.86869549s FAIL (2025-07-18 14:53:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.868724705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.925536978s FAIL (2025-07-18 14:52:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.925563999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 26.641395344s FAIL (2025-07-18 14:51:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 26.641434198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.827118178s FAIL (2025-07-18 14:50:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.827144708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.298321496s FAIL (2025-07-18 14:49:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.298348779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 13.368682942s FAIL (2025-07-18 14:48:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.368712159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.020611128s FAIL (2025-07-18 14:48:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.020646546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 25.50568015s FAIL (2025-07-18 14:48:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.50570657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.009413235s FAIL (2025-07-18 14:46:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.0094374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 25.390397848s FAIL (2025-07-18 14:46:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 25.390421732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 13.154480361s FAIL (2025-07-18 14:45:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.154512833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.78840714s FAIL (2025-07-18 14:43:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.788434854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.438129815s FAIL (2025-07-18 14:43:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.438154883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.625970635s FAIL (2025-07-18 14:42:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.625999901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 23.764422134s FAIL (2025-07-18 14:41:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.764448845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 14.519325581s FAIL (2025-07-18 14:40:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.519358714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.952533571s FAIL (2025-07-18 14:39:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.952552457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.245416591s FAIL (2025-07-18 14:39:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== KdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Jul 18 14:38:10.476236479 (605h27m45.246877844s) [seg_pos={0 0}, upload=484203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 14:38:10.476300692 (605h27m45.246942027s) [upload=484203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 14:38:10.476304409 (605h27m45.246945744s) [upload=484203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 14:38:10.759796487 (605h27m45.530437822s) [upload=484203]: Done uploading segments... Jul 18 14:38:10.759814852 (605h27m45.530456187s) [upload=484203]: Upload finished. err: <nil> Jul 18 14:38:57.183367404 (605h28m31.954008739s) [upload=484215]: Starting upload Jul 18 14:38:57.183435405 (605h28m31.95407674s) [upload=484215]: Done starting upload Jul 18 14:38:57.183531618 (605h28m31.954172944s) [upload=484215]: Uploading segments... Jul 18 14:38:57.183597846 (605h28m31.954239181s) [upload=484215]: Got next segment. Inline: false Jul 18 14:38:57.183629236 (605h28m31.954270571s) [upload=484215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 14:38:57.183633654 (605h28m31.954274989s) [upload=484215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 14:38:57.423199125 (605h28m32.19384046s) [seg_pos={0 0}, upload=484215]: Begin upload segment... Jul 18 14:38:57.423243880 (605h28m32.193885215s) [seg_pos={0 0}, upload=484215]: Done begin upload segment. Jul 18 14:38:57.423268667 (605h28m32.193909992s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423301781 (605h28m32.193943116s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423651430 (605h28m32.194292765s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423770988 (605h28m32.194412323s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.423916057 (605h28m32.194557432s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424185402 (605h28m32.194826737s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424508129 (605h28m32.195149464s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.424783546 (605h28m32.195424881s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425063291 (605h28m32.195704626s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425330843 (605h28m32.195972178s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425566795 (605h28m32.19620813s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.425771055 (605h28m32.19641239s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426099003 (605h28m32.196740338s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426492145 (605h28m32.19713348s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426719891 (605h28m32.197361226s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:57.426916066 (605h28m32.197557401s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Uploading piece... Jul 18 14:38:59.195917191 (605h28m33.966558516s) [upload=484215]: Next returned nil segment Jul 18 14:38:59.195938781 (605h28m33.966580116s) [upload=484215]: Waiting for error group managing segments... Jul 18 14:38:59.486814738 (605h28m34.257456073s) [noise=true , address=45.91.116.7:4011 , piece_id=UZZHVBINUINKGTIGG22MILT6A7IER3TBDCIXQS6X4YLJWKEPESHA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.494719673 (605h28m34.265361008s) [noise=true , address=45.91.116.12:4226 , piece_id=TP4OXGMKMUFGMDIQIK7PKYTGG4AIH3ORXK6RBJEW3JBOIMVDEGUQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.499194715 (605h28m34.26983605s) [noise=true , address=45.91.116.6:4974 , piece_id=65WQWYLHS4XJV7CO5VSX2OOIUEWYMB6SCCCFQVZ4PSD6EMTR5ABQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.514471424 (605h28m34.285112759s) [noise=true , address=45.91.116.13:4155 , piece_id=X7DPMK2NQJAMI6ZH2CGFM354TVUILOLKCUFNMRTE2AHCBPRDSLFQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.569387026 (605h28m34.340028361s) [noise=true , address=45.91.116.5:4298 , piece_id=2XPWPOKJIPZYCND2CKNAHKJEMNNG2BWSZHVVVAATQXIZLBSIGGDQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.584150324 (605h28m34.354791659s) [noise=true , address=45.91.116.9:4263 , piece_id=HIXKJVLIZFC5A3JFJRR6SSY2TTTWUIF4SK5THMHGFIVYM6K27UYQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.611162827 (605h28m34.381804162s) [noise=true , address=104.36.82.11:4113 , piece_id=MQZ4ETHXS2YIHLMDY2FGJHVUAA37VHUD77HWJYXTJARNQ2WUTGTA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:38:59.631692548 (605h28m34.402333883s) [noise=true , address=45.91.116.11:4119 , piece_id=NK666VRWFNT6M7NXIZH62V3BSA5SBS52PJ3KVL5ARTWCJKOTE3RQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.016455232 (605h28m34.787096567s) [noise=true , address=45.91.116.7:4009 , piece_id=5DRDFXE7LSTLBI4VPIYGGV5DQMLSU42ATUUGXAFFP7NFEYXTFB3A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:00.534448649 (605h28m35.305089984s) [noise=true , address=45.91.116.11:4117 , piece_id=FQWXQGB5IXSRXKHAWK6KI5IC4CMDQDBER664W37IS5FABAMMPOIQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:19.552072796 (605h28m54.322714121s) [noise=true , address=77.90.31.11:4555 , piece_id=TTVU3CQYWBVVW5GCRPCHS6JY534FJX43IPY6JZ7FCLMIWBUNSZHQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:24.814185111 (605h28m59.584826446s) [noise=true , address=77.90.31.12:4592 , piece_id=XYIQ2SWMI26TSLXAQMUVQ6RHOZNBSKJ4QFPRTNDQRSGTQNOWZ2YQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: <nil> Jul 18 14:39:26.545214605 (605h29m1.31585593s) [noise=true , address=77.90.31.13:4626 , piece_id=7DAIENYWREBFIDJJPZXRPTPNOLS4LNTFVNLYTL74JK464V5P3DZQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545271163 (605h29m1.315912498s) [noise=true , address=77.90.31.17:4772 , piece_id=M3ENQJ75WLD47R5B64ABC2UOREGNMNGBXUBGMRABHV6VAECDFN4A, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 18 14:39:26.545280089 (605h29m1.315921425s) [noise=true , address=77.90.31.12:4590 , piece_id=TLU6NAKNHTI6EITB2DV4XO3OWQ7EXJ4Y5OT6WXC5XIGC4RVHKXRA, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43108->77.90.31.12:4590: use of closed network connection Jul 18 14:39:26.545375933 (605h29m1.316017268s) [noise=true , address=77.90.31.19:4843 , piece_id=7EQMO34LB6JPC3MXGM4OBTTEIDD2W4GE2L5DMNX5SLCABGDNOHRQ, seg_pos={0 0}, upload=484215]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52640->77.90.31.19:4843: use of closed network connection Jul 18 14:39:26.545395971 (605h29m1.316037306s) [seg_pos={0 0}, upload=484215]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 14:39:26.789355901 (605h29m1.559997236s) [upload=484215]: Done uploading segments... Jul 18 14:39:26.789378063 (605h29m1.560019398s) [upload=484215]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.245434957s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.139629839s FAIL (2025-07-18 09:40:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.139657502s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.023322329s FAIL (2025-07-18 09:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.023344431s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.054959684s FAIL (2025-07-18 09:38:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.054980115s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.860981527s FAIL (2025-07-18 09:37:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.861003048s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.953493255s FAIL (2025-07-18 09:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.953514295s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.221367088s FAIL (2025-07-18 09:35:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.221389763s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.961028413s FAIL (2025-07-18 09:35:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.961049674s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.252209183s FAIL (2025-07-18 09:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.252228711s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.512918926s FAIL (2025-07-18 09:33:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.512939214s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.449454335s FAIL (2025-07-18 09:33:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.449476758s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.965859758s FAIL (2025-07-18 09:31:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.965883544s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.463871777s FAIL (2025-07-18 09:31:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.463895714s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.236205856s FAIL (2025-07-18 09:31:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.236227398s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)