Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.........X................] duration: 6.381187522s FAIL (2025-09-27 10:22:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.38122318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.983203949s FAIL (2025-09-27 10:22:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.983231451s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.........X...............] duration: 24.59725581s FAIL (2025-09-27 10:22:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 24.597290314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.170602258s FAIL (2025-09-27 10:21:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.170632576s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.........X..............] duration: 6.188238471s FAIL (2025-09-27 10:20:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.188273308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.108612743s FAIL (2025-09-27 10:20:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.108634184s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.........X.............] duration: 6.34731316s FAIL (2025-09-27 10:20:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.347346554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.262934244s FAIL (2025-09-27 10:20:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.262979109s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.........X............] duration: 6.070077716s FAIL (2025-09-27 10:19:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.070108785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.282166261s FAIL (2025-09-27 10:18:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 7.28219729s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.........X...........] duration: 6.114793373s FAIL (2025-09-27 10:18:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.114828348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.19199844s FAIL (2025-09-27 10:18:23 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.192031963s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.........X..........] duration: 6.089741114s FAIL (2025-09-27 10:18:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.089788013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.918777161s FAIL (2025-09-27 10:17:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.918805716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.........X.........] duration: 6.27565302s FAIL (2025-09-27 10:17:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.275687886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.601147457s FAIL (2025-09-27 10:17:21 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.601192713s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.........X........] duration: 6.552674588s FAIL (2025-09-27 10:16:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.552707181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.248937458s FAIL (2025-09-27 10:16:30 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.248968226s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration: 7.738212141s FAIL (2025-09-27 10:15:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.738244382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.946654866s FAIL (2025-09-27 10:15:30 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.946687718s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.011714043s FAIL (2025-09-27 10:14:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.01182307s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................................X.........X......] duration: 6.247783678s FAIL (2025-09-27 10:14:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.247814576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.330171548s FAIL (2025-09-27 10:14:08 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.330231842s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................................X.........X.....] duration: 13.387068523s FAIL (2025-09-27 10:13:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 13.387096145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.362145902s FAIL (2025-09-27 10:13:11 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.362179387s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................................X.........X....] duration: 6.573257085s FAIL (2025-09-27 10:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.57329155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.96787945s FAIL (2025-09-27 10:11:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.96791662s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................................X.........X...] duration: 6.271765148s FAIL (2025-09-27 10:11:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.271796918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.397129483s FAIL (2025-09-27 10:11:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.397166582s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........................................X.........X..] duration: 6.216647597s FAIL (2025-09-27 10:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.216679569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.364781313s FAIL (2025-09-27 10:11:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.364806831s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.332326631s FAIL (2025-09-27 10:10:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.3323573s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................................X.........X.] duration: 6.030794741s FAIL (2025-09-27 10:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.030923947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.788703548s FAIL (2025-09-27 10:09:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.788732844s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................................X.........X] duration: 30.310648361s FAIL (2025-09-27 10:09:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 10:08:54.701836491 (92h37m40.163928964s) [upload=81506]: Starting upload Sep 27 10:08:54.701907486 (92h37m40.163999969s) [upload=81506]: Done starting upload Sep 27 10:08:54.702048706 (92h37m40.164141179s) [upload=81506]: Uploading segments... Sep 27 10:08:54.702171569 (92h37m40.164264042s) [upload=81506]: Got next segment. Inline: false Sep 27 10:08:54.702208850 (92h37m40.164301323s) [upload=81506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:08:54.702215923 (92h37m40.164308406s) [upload=81506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:08:54.969166614 (92h37m40.431259087s) [seg_pos={0 0}, upload=81506]: Begin upload segment... Sep 27 10:08:54.969232890 (92h37m40.431325363s) [seg_pos={0 0}, upload=81506]: Done begin upload segment. Sep 27 10:08:54.969257367 (92h37m40.43134985s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969283888 (92h37m40.431376371s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969340405 (92h37m40.431432888s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.969978390 (92h37m40.432070873s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970395605 (92h37m40.432488088s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.970784916 (92h37m40.432877389s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971196520 (92h37m40.433288993s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.971597925 (92h37m40.433690398s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972102735 (92h37m40.434195208s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.972573262 (92h37m40.434665735s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.973043978 (92h37m40.435136461s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974441079 (92h37m40.436533562s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.974578891 (92h37m40.436671374s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976000609 (92h37m40.438093092s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.976120446 (92h37m40.438212929s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:54.982203880 (92h37m40.444296354s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Uploading piece... Sep 27 10:08:56.289285223 (92h37m41.751377706s) [noise=true , address=77.90.31.19:4841 , piece_id=WL32HJMZ5RXCERVELIZRR44NHVOQXMHSCGH7K2V7RAHCCMLO7PXA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:46714->77.90.31.19:4841: write: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:08:56.673929411 (92h37m42.136021884s) [upload=81506]: Next returned nil segment Sep 27 10:08:56.673951022 (92h37m42.136043505s) [upload=81506]: Waiting for error group managing segments... Sep 27 10:08:56.955200552 (92h37m42.417293035s) [noise=true , address=145.79.227.7:4294 , piece_id=AWYJWR2KKINRU4SE47HZCBAI4CS4JUVH43U57GCU6VCVPSOVSRLQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.956368125 (92h37m42.418460598s) [noise=true , address=145.79.227.4:4185 , piece_id=6YB52H4IQLXSQ2G4XVE5672IDQKNGJQBMXH5DWZ5RX3L64RCWH2Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:56.978191733 (92h37m42.440284216s) [noise=true , address=145.79.227.3:4149 , piece_id=KV3GVFQZX5CUMZHDPIK7ULTUAQIK3T6MYF5TPFKVLFOQ7NH7HMCA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.006811773 (92h37m42.468904246s) [noise=true , address=145.79.227.16:4911 , piece_id=BEOQVC5R2EC3C44ZE52XU7XENJ2DQ3XFTYY53QHIAT4QOW7NKO7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.072636459 (92h37m42.534728942s) [noise=true , address=45.91.116.5:4299 , piece_id=IBUOE7ERFYEBGJGT75RD2INDBDINPEDXCQMXJIAUQGTJP524V53Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.120035119 (92h37m42.582127592s) [noise=true , address=45.91.116.14:4189 , piece_id=HQC6B7SBEQYXRN257XIIFNFU77CODY3TI5KVEIXSAAWESBOQ365Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.126381714 (92h37m42.588474187s) [noise=true , address=45.91.116.9:4263 , piece_id=JVOT6ZUQJL6CGBXBFLRJVNZNXC54A7X7ILKHYXYICTXQEIG45CFA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.137709205 (92h37m42.599801678s) [noise=true , address=45.91.116.4:4938 , piece_id=3ZJDZS2KI4YVFE5M7LRRVA7F2T2WVFYE3XARYZRGBZCA3ZV4UKDA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.142137266 (92h37m42.604229739s) [noise=true , address=45.91.116.7:4011 , piece_id=KUPAIMXPQU3IJPQFTYDKLV3USXCH6GYXNNHXTQ4YPQOCHMYOSA7Q, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.164328814 (92h37m42.626421287s) [noise=true , address=45.91.116.11:4118 , piece_id=QOZAW5EWGS5NFIUOQNV65ATYUFRD5JPBSW55DTC2T73QGGDZCT7A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:08:57.180695641 (92h37m42.642788125s) [noise=true , address=45.91.116.13:4153 , piece_id=I2OB346NXB4BGOYQNDRNVPFAHBE3MNEHIKOBGOQZBC7AFV4EH36A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: <nil> Sep 27 10:09:23.967034962 (92h38m9.429127445s) [noise=true , address=77.90.31.21:4879 , piece_id=ON4TU6PWIX2GYTNW47FN5P43JHWWDOR24T2WHTNU7HSBQUXUTOJQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:40444->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967241595 (92h38m9.429334089s) [noise=true , address=77.90.31.11:4554 , piece_id=O53R3SNINEUVUOMMQPQGJS5EJK4QQCMSS6AKAKZ73AVVWLUXFCGA, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: rpc: dial tcp 77.90.31.11:4554: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967433832 (92h38m9.429526315s) [noise=true , address=77.90.31.12:4592 , piece_id=X7LYLLRGDHNKHEVDOJARRZIFFKOZZHJJ3VASB22SXTL6WPIAJYIQ, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:50034->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:09:23.967581102 (92h38m9.429673575s) [noise=true , address=77.90.31.18:4805 , piece_id=DSXOKC5I7QNHLC2ABUAZZLEHWS3BW52CAA35VE3N62WYZHDLJQ3A, seg_pos={0 0}, upload=81506]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: context deadline exceeded; context deadline exceeded Sep 27 10:09:23.967608064 (92h38m9.429700547s) [seg_pos={0 0}, upload=81506]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 10:09:24.275654922 (92h38m9.737747405s) [upload=81506]: Done uploading segments... Sep 27 10:09:24.275692313 (92h38m9.737784796s) [upload=81506]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.310673449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.145794419s FAIL (2025-09-27 10:08:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.145891353s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.822882038s FAIL (2025-09-27 10:08:32 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.822916183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.972176966s FAIL (2025-09-27 10:08:17 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.97221096s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 6.732246601s FAIL (2025-09-27 10:07:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 6.732302107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.202541038s FAIL (2025-09-27 10:07:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.202575284s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.977831291s FAIL (2025-09-27 10:07:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.977908428s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................................X........] duration: 6.973707031s FAIL (2025-09-27 10:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 6.973741066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.227019271s FAIL (2025-09-27 10:06:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.227048136s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................................X.......] duration: 6.276311438s FAIL (2025-09-27 10:06:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 6.276336997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.000287884s FAIL (2025-09-27 10:06:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.000326769s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 6.111193018s FAIL (2025-09-27 10:05:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 6.11122999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.086321799s FAIL (2025-09-27 10:05:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.086356826s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 5.967229582s FAIL (2025-09-27 10:04:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 5.967260741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.005774108s FAIL (2025-09-27 10:04:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.005806499s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................................X....] duration: 6.198862339s FAIL (2025-09-27 10:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 27 10:00:38.420650461 (92h29m23.882742944s) [upload=81388]: Starting upload Sep 27 10:00:38.420793854 (92h29m23.882886347s) [upload=81388]: Done starting upload Sep 27 10:00:38.420976371 (92h29m23.883068854s) [upload=81388]: Uploading segments... Sep 27 10:00:38.421271303 (92h29m23.883363786s) [upload=81388]: Got next segment. Inline: false Sep 27 10:00:38.421335736 (92h29m23.883428209s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:00:38.421341376 (92h29m23.883433849s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:00:38.681739945 (92h29m24.143832428s) [seg_pos={0 0}, upload=81388]: Begin upload segment... Sep 27 10:00:38.681857047 (92h29m24.14394952s) [seg_pos={0 0}, upload=81388]: Done begin upload segment. Sep 27 10:00:38.681906251 (92h29m24.143998734s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.681922452 (92h29m24.144014925s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682555497 (92h29m24.14464798s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.682753274 (92h29m24.144845757s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683064627 (92h29m24.14515711s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683157924 (92h29m24.145250397s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683672574 (92h29m24.145765047s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.683691601 (92h29m24.145784084s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684073859 (92h29m24.146166342s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684219837 (92h29m24.14631232s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.684524297 (92h29m24.14661677s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685148405 (92h29m24.147240888s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685733478 (92h29m24.147825962s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.685851864 (92h29m24.147944348s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.687897009 (92h29m24.149989492s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:38.689055755 (92h29m24.151148238s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Uploading piece... Sep 27 10:00:40.282610260 (92h29m25.744702743s) [upload=81388]: Next returned nil segment Sep 27 10:00:40.282633715 (92h29m25.744726198s) [upload=81388]: Waiting for error group managing segments... Sep 27 10:00:40.662144810 (92h29m26.124237293s) [noise=true , address=145.79.227.4:4185 , piece_id=LZNZHVTRJLWE6RQLOKU7M2X7IAYATJC5QAD7AT6WTGSK33O726BA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.662707381 (92h29m26.124799854s) [noise=true , address=145.79.227.14:4839 , piece_id=PRRXFRADD7HFYJY7SO35EQNMB6L7TWKMRILMD6E7NWSZ7MCEKUPA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.683443827 (92h29m26.14553631s) [noise=true , address=145.79.227.7:4293 , piece_id=KBB6EYG2TOGN53V7V7U4CGF5YV6OMCHFICT4J74MWDYW3EHR5BOQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693070869 (92h29m26.155163352s) [noise=true , address=145.79.227.15:4873 , piece_id=G5P7LS7JKE36BJ5E3WEGADDVORBLXKBIABSCEZTG3E5IDHEHR5XA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.693568256 (92h29m26.155660739s) [noise=true , address=145.79.227.13:4803 , piece_id=FDOVBMZGKWIUGKYWXLFVAPQP52JFJPKAVFP3XEIJFQGNZTRTAXSQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.718910152 (92h29m26.181002625s) [noise=true , address=145.79.227.12:4765 , piece_id=3UFH6N3P6F3ZADQJZJ5BUTJUMBWSBTH6UZ56KGJSWCZ7JBBKZXTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.854446550 (92h29m26.316539033s) [noise=true , address=77.90.31.17:4772 , piece_id=2DW7RGX6ANBQFFOHCZ54WVUZGO23FBSHFDTIA26DXYB6M4QHW2KQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:40.878422957 (92h29m26.34051544s) [noise=true , address=77.90.31.15:4698 , piece_id=CK4353JGA6IZWNLFAAN7H5DS4CBQKW42RVXLC5GZFCBBJFMM7CUA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:41.833708360 (92h29m27.295800853s) [noise=true , address=77.90.31.14:4663 , piece_id=YFNCGWFT6GEYLA7JJPD3PXTGZGNYBCVTSI2QVMINFCPEDW2LNCNQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.006488081 (92h29m33.468580564s) [noise=true , address=45.91.116.4:4939 , piece_id=QFLURPM5VXOBN3WAD752L4C776UTVCPJ3VJYOCLU4EEHA6WX7TTQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.370051668 (92h29m33.832144142s) [noise=true , address=45.91.116.8:4048 , piece_id=IAHVYMCQA2HQVNVIJA4X23RFGINGF6VQSJQWUWSUAB7A44F2TXOA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.431521188 (92h29m33.893613671s) [noise=true , address=45.91.116.5:4297 , piece_id=47MFKK4X7SBX6QYS2Y5AUEF4NARLNK54JRQODG2VU72V63WPLVYQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449266640 (92h29m33.911359123s) [noise=true , address=45.91.116.12:4226 , piece_id=XMBFPDJRT3PA7RBYT2XF67J7G7S4WQDQMJGLMQXE247ENJ4SWHLA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: <nil> Sep 27 10:00:48.449310673 (92h29m33.911403157s) [seg_pos={0 0}, upload=81388]: Segment reached optimal threshold of 13 pieces. Sep 27 10:00:48.449529239 (92h29m33.911621732s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449544729 (92h29m33.911637212s) [noise=true , address=45.91.116.10:4083 , piece_id=YQVWGWA5ZU6C2XOP6TJEOTIQ2CLHTXAQACCO6G2B773JSUTYBXHA, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449590616 (92h29m33.91168311s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56810->45.91.116.9:4262: use of closed network connection Sep 27 10:00:48.449600485 (92h29m33.911692968s) [noise=true , address=45.91.116.9:4262 , piece_id=WZTGG7QLQ2TBCVIRQOGYRIVZ4UMIA4QLPGAC2KOHV7YT363NIWFQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449662714 (92h29m33.911755207s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Sep 27 10:00:48.449672783 (92h29m33.911765266s) [noise=true , address=45.91.116.4:4940 , piece_id=Q64C34YDTFBXTIOTQGKVPHN3N7XY4KMVVMLCKLN5FQCNQI3DSVIQ, seg_pos={0 0}, upload=81388]: Long tail cancellation due to optimal threshold Sep 27 10:00:48.449713340 (92h29m33.911805823s) [seg_pos={0 0}, upload=81388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 27 10:00:48.449928981 (92h29m33.912021464s) [upload=81388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:00:48.449937136 (92h29m33.912029619s) [upload=81388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:00:48.829760556 (92h29m34.291853049s) [upload=81388]: Done uploading segments... Sep 27 10:00:48.829788639 (92h29m34.291881122s) [upload=81388]: Upload finished. err: <nil> Duration: 6.198891765s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)