Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [........X.X..X.....X......................X.................] duration: 2.168173744s FAIL (2025-08-16 15:59:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.168197728s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.29665463s FAIL (2025-08-16 15:59:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 16 15:58:37.346150687 (190h32m31.777230818s) [upload=198220]: Starting upload Aug 16 15:58:37.346198148 (190h32m31.777278269s) [upload=198220]: Done starting upload Aug 16 15:58:37.346385828 (190h32m31.777465959s) [upload=198220]: Uploading segments... Aug 16 15:58:37.346416075 (190h32m31.777496207s) [upload=198220]: Got next segment. Inline: false Aug 16 15:58:37.346447866 (190h32m31.777527997s) [upload=198220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:58:37.346452154 (190h32m31.777532285s) [upload=198220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:58:37.817597117 (190h32m32.248677238s) [seg_pos={0 0}, upload=198220]: Begin upload segment... Aug 16 15:58:37.817640049 (190h32m32.24872018s) [seg_pos={0 0}, upload=198220]: Done begin upload segment. Aug 16 15:58:37.817661690 (190h32m32.248741821s) [noise=true , address=104.36.82.3:4783 , piece_id=QJDIMTDNXO3MZKL6SLVRYSFJQCTK6UEJ2K75MPEZG6F5UUCJQATQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.817671740 (190h32m32.248751871s) [noise=true , address=77.90.31.12:4617 , piece_id=WIPOE6CXOPL2XFRENW5N5PCZOXRWQYBP7HLWTAALJXWABGCX2EIA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.817738237 (190h32m32.248818368s) [noise=true , address=66.23.193.175:4560 , piece_id=6ZBNVG2QQ2KKVJYVADDU6UDZ4LQ4QISC4YP5RQYB354GGA47BZPA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818022892 (190h32m32.249103023s) [noise=true , address=104.36.82.60:4359 , piece_id=2QIVQ7BDNCMF4XGCABJLNC3DKORDB4KNTXS3XMZDASKJ5J7VTCLA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818153852 (190h32m32.249233983s) [noise=true , address=45.91.116.4:4966 , piece_id=GOBKUVEBZTQBRJUJV6TFUWT6R62KRLLVTDA744IBBANGSF4GTUKQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818290754 (190h32m32.249370885s) [noise=true , address=104.36.82.44:4902 , piece_id=ABKZIRLSLWZQNDR3RHQUTNAPVHKST4XRBJRGEC3CDTK4A6XS43MA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818456681 (190h32m32.249536812s) [noise=true , address=104.36.82.36:4646 , piece_id=OB4M2WNYDDGJELHJELBCEX37LP4PLZHMQ3EQCXR254V5DDJSCPQQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818635784 (190h32m32.249715915s) [noise=true , address=79.127.220.101:4061 , piece_id=BPGFZUHCVZ45GSW36KV2BYUM4ZXRA65JCZJROVQVBNUPOHSVTDXQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.818910500 (190h32m32.249990631s) [noise=true , address=77.90.31.11:4580 , piece_id=CX6J6DBAYZIF7JDBZ65GGK4TDEYECPGYCUI6FKLPK2FTMOTLVCMQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819159436 (190h32m32.250239567s) [noise=true , address=104.36.82.59:4320 , piece_id=F3HBFLM2M5IK2ZSO56F6F55FHEG2XGP6BIFVA2GTYL6N7T5GOI5A, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819230332 (190h32m32.250310463s) [noise=true , address=104.36.82.34:4560 , piece_id=ACTST5OOAHDBF2USKQHPRW2P2EVLCDCACAETDGP4I56MVPDNTJMQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819382403 (190h32m32.250462534s) [noise=true , address=45.91.116.11:4125 , piece_id=AECHS63XG7UNNQZTUO7LWBIGUZK2OCHKASXJYHGVGADN3BBVZDJA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819489357 (190h32m32.250569489s) [noise=true , address=79.127.201.215:4351 , piece_id=JKUXCERASDBBOPOLXAA26TAXQRXZDA3BWG7RTHJRMZWEZZ6LLEFA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819638383 (190h32m32.250718514s) [noise=true , address=104.36.82.37:4676 , piece_id=NV6EOFKYSW224TZYWDB6KBL76GSGCDCKDHWL5DGK473Q725XTDOQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819735298 (190h32m32.250815429s) [noise=true , address=77.90.31.18:4812 , piece_id=VOZBN5CSBZUVK2MC3FXWNMNPKRAH66SMJJZYJHZ7XVPN7646YGQA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.819843696 (190h32m32.250923827s) [noise=true , address=77.90.31.15:4718 , piece_id=GYP75A4VAMOIJMNQLOXWZG7GPTWJ3IZ7624UUETEUZO3EEPKXUOA, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.820035162 (190h32m32.251115293s) [noise=true , address=77.90.31.13:4637 , piece_id=RVG5QQ7KVUWG4BU3HRW2DEBY55A4Y464OECTTSCY34TEAGNQZD2A, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:37.820219655 (190h32m32.251299786s) [noise=true , address=77.90.31.10:4549 , piece_id=TZVUJM6632A6647UWVLM724UNEHGDW3BIOEVKAQ3AM3WMZSLOTCQ, seg_pos={0 0}, upload=198220]: Uploading piece... Aug 16 15:58:39.011297219 (190h32m33.44237734s) [upload=198220]: Next returned nil segment Aug 16 15:58:39.011314292 (190h32m33.442394413s) [upload=198220]: Waiting for error group managing segments... Aug 16 15:58:40.704879809 (190h32m35.13595994s) [noise=true , address=77.90.31.12:4617 , piece_id=WIPOE6CXOPL2XFRENW5N5PCZOXRWQYBP7HLWTAALJXWABGCX2EIA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:40.782952212 (190h32m35.214032343s) [noise=true , address=77.90.31.13:4637 , piece_id=RVG5QQ7KVUWG4BU3HRW2DEBY55A4Y464OECTTSCY34TEAGNQZD2A, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:40.825373139 (190h32m35.25645327s) [noise=true , address=77.90.31.10:4549 , piece_id=TZVUJM6632A6647UWVLM724UNEHGDW3BIOEVKAQ3AM3WMZSLOTCQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:40.842627795 (190h32m35.273707926s) [noise=true , address=77.90.31.15:4718 , piece_id=GYP75A4VAMOIJMNQLOXWZG7GPTWJ3IZ7624UUETEUZO3EEPKXUOA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:41.221839414 (190h32m35.652919545s) [noise=true , address=77.90.31.18:4812 , piece_id=VOZBN5CSBZUVK2MC3FXWNMNPKRAH66SMJJZYJHZ7XVPN7646YGQA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:41.505850876 (190h32m35.936930997s) [noise=true , address=79.127.220.101:4061 , piece_id=BPGFZUHCVZ45GSW36KV2BYUM4ZXRA65JCZJROVQVBNUPOHSVTDXQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:42.176701805 (190h32m36.607781926s) [noise=true , address=45.91.116.4:4966 , piece_id=GOBKUVEBZTQBRJUJV6TFUWT6R62KRLLVTDA744IBBANGSF4GTUKQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:43.255852950 (190h32m37.686933081s) [noise=true , address=45.91.116.11:4125 , piece_id=AECHS63XG7UNNQZTUO7LWBIGUZK2OCHKASXJYHGVGADN3BBVZDJA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:49.381258276 (190h32m43.812338397s) [noise=true , address=79.127.201.215:4351 , piece_id=JKUXCERASDBBOPOLXAA26TAXQRXZDA3BWG7RTHJRMZWEZZ6LLEFA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:58:49.921371631 (190h32m44.352451762s) [noise=true , address=66.23.193.175:4560 , piece_id=6ZBNVG2QQ2KKVJYVADDU6UDZ4LQ4QISC4YP5RQYB354GGA47BZPA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: <nil> Aug 16 15:59:36.665595078 (190h33m31.096675209s) [noise=true , address=104.36.82.60:4359 , piece_id=2QIVQ7BDNCMF4XGCABJLNC3DKORDB4KNTXS3XMZDASKJ5J7VTCLA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV, address:104.36.82.60:4359): protocol: rpc: dial tcp 104.36.82.60:4359: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665630385 (190h33m31.096710516s) [noise=true , address=104.36.82.3:4783 , piece_id=QJDIMTDNXO3MZKL6SLVRYSFJQCTK6UEJ2K75MPEZG6F5UUCJQATQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: rpc: dial tcp 104.36.82.3:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665688396 (190h33m31.096768527s) [noise=true , address=104.36.82.59:4320 , piece_id=F3HBFLM2M5IK2ZSO56F6F55FHEG2XGP6BIFVA2GTYL6N7T5GOI5A, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:12HrqyHVr44byroEByGHdTQyma8kYkh5HBqLocKCGEMFM5teqfb, address:104.36.82.59:4320): protocol: write tcp 188.166.225.170:40270->104.36.82.59:4320: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665717712 (190h33m31.096797833s) [noise=true , address=104.36.82.37:4676 , piece_id=NV6EOFKYSW224TZYWDB6KBL76GSGCDCKDHWL5DGK473Q725XTDOQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:18yqKh7gMSrXWavfChEJRX7tV5rS9xTSo4SbmgPMtnYKfgbvF8, address:104.36.82.37:4676): protocol: write tcp 188.166.225.170:39122->104.36.82.37:4676: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665738021 (190h33m31.096818152s) [noise=true , address=104.36.82.44:4902 , piece_id=ABKZIRLSLWZQNDR3RHQUTNAPVHKST4XRBJRGEC3CDTK4A6XS43MA, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665749833 (190h33m31.096829964s) [noise=true , address=104.36.82.36:4646 , piece_id=OB4M2WNYDDGJELHJELBCEX37LP4PLZHMQ3EQCXR254V5DDJSCPQQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:1zA8TqySnmGc9bVQnNNMSZEPbv9Gf1Wa1BwYrKjc2ZGgtyEmrh, address:104.36.82.36:4646): protocol: rpc: dial tcp 104.36.82.36:4646: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:59:36.665762618 (190h33m31.096842749s) [noise=true , address=77.90.31.11:4580 , piece_id=CX6J6DBAYZIF7JDBZ65GGK4TDEYECPGYCUI6FKLPK2FTMOTLVCMQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, address:77.90.31.11:4580): protocol: rpc: dial tcp 77.90.31.11:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665773839 (190h33m31.09685397s) [noise=true , address=104.36.82.34:4560 , piece_id=ACTST5OOAHDBF2USKQHPRW2P2EVLCDCACAETDGP4I56MVPDNTJMQ, seg_pos={0 0}, upload=198220]: Done uploading piece. err: ecclient: upload failed (node:12sNZ8qhPusEuj4saH5puqvuptETQfx8zJQ4SbsRVT48K2WRx58, address:104.36.82.34:4560): protocol: write tcp 188.166.225.170:54028->104.36.82.34:4560: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:36.665793307 (190h33m31.096873438s) [seg_pos={0 0}, upload=198220]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:59:36.960025406 (190h33m31.391105547s) [upload=198220]: Done uploading segments... Aug 16 15:59:36.960055193 (190h33m31.391135324s) [upload=198220]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.296667435s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...............X......X..XXXXXXXXXX.XX.XXXXXX......X.......X] duration: 30.228443741s FAIL (2025-08-16 15:59:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:59:06.394943718 (190h33m0.826023849s) [upload=198229]: Starting upload Aug 16 15:59:06.395032928 (190h33m0.82611306s) [upload=198229]: Done starting upload Aug 16 15:59:06.395082964 (190h33m0.826163085s) [upload=198229]: Uploading segments... Aug 16 15:59:06.395189999 (190h33m0.82627013s) [upload=198229]: Got next segment. Inline: false Aug 16 15:59:06.395223944 (190h33m0.826304085s) [upload=198229]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:59:06.395228413 (190h33m0.826308554s) [upload=198229]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:59:06.842640526 (190h33m1.273720647s) [seg_pos={0 0}, upload=198229]: Begin upload segment... Aug 16 15:59:06.842675673 (190h33m1.273755804s) [seg_pos={0 0}, upload=198229]: Done begin upload segment. Aug 16 15:59:06.842705610 (190h33m1.273785731s) [noise=true , address=77.90.31.11:4555 , piece_id=5MM5BLXB53ECYEYFRR7F7PMK4OSUND3C63MEEXB4U72HWISRQJIQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.842789090 (190h33m1.273869211s) [noise=true , address=77.90.31.16:4734 , piece_id=K4AHIZJG4LIAEZMEGHRVLTIQBCZRL666M4N5LRQ2U4URRH2VVE7A, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843117278 (190h33m1.274197419s) [noise=true , address=45.91.116.13:4155 , piece_id=345OQ37OEAQOKMVZN6SFC4UDXPQO57OD2F2DK4WHHQ7KZ55YKUIQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843357989 (190h33m1.27443812s) [noise=true , address=45.91.116.8:4048 , piece_id=74SZV342HX3KGO7KUJK5HH2TNRB2QHSHUL6EWVMYR3MGKLJ3BSQQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843412914 (190h33m1.274493045s) [noise=true , address=104.36.82.13:4185 , piece_id=ZTLWEEWTXK7ES3SP2DXAZHGQVT3APRTE24VAAOYOG5UYAZLWRLFA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843690104 (190h33m1.274770235s) [noise=true , address=45.91.116.5:4297 , piece_id=HXOBR2IKERVXURVJMELFKAJ7IHWJHDNFDTZTOKQIM3TITRBE3CYA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843769666 (190h33m1.274849797s) [noise=true , address=104.36.82.2:4730 , piece_id=AKGJIR4BVKO2S5BR5BQO7FX5ZTG3LV62C6PC2FKPT3BRTEDPSDSA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.843980791 (190h33m1.275060922s) [noise=true , address=104.36.82.7:4912 , piece_id=UHSAC26VUMU4O5I23DYOHAWZOSAYHAB6DAF7U3W6KNHTIIX7EV7Q, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844084709 (190h33m1.27516484s) [noise=true , address=45.91.116.6:4973 , piece_id=GHAZQGUX7S6NNNJT73QEURUTU43D4PR7MY2DIKHWXNIHFIUU2EWA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844293839 (190h33m1.27537397s) [noise=true , address=77.90.31.14:4661 , piece_id=OTLHFI2LFB2Z2NYFTJISFHRGQUJ6XHCA32GRM7WF6BZ463543UOQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844501106 (190h33m1.275581237s) [noise=true , address=104.36.82.12:4149 , piece_id=OLBI5RRDUMHYQ7522VEHWDMRAGNKWTEBAZKMJGBY3WBY4SMZDQCA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844542775 (190h33m1.275622907s) [noise=true , address=45.91.116.6:4975 , piece_id=VKSCDFJILOCFYXTB3OFPTTXZBUQ3W74VS7GTUQIVWYVITJC3TCGQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844748810 (190h33m1.275828941s) [noise=true , address=45.91.116.14:4189 , piece_id=IYI6IFI5MI5GFVVMTHJV6WCANYULN2C2VEHLC2DE3LUA7QQKD62Q, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.844769108 (190h33m1.275849239s) [noise=true , address=77.90.31.13:4627 , piece_id=PWNUKSQLM54WA3ZMHMWGSG5XGJ67A7QVGC4G5LEQT3TVFY6DCWQA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.845019627 (190h33m1.276099759s) [noise=true , address=77.90.31.15:4699 , piece_id=VFGEO5AOPAUNWY5KITKM6PZNJXMPPPZDNZ4XLHBLZY3RTPZM6IAA, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:06.845085103 (190h33m1.276165234s) [noise=true , address=45.91.116.9:4263 , piece_id=Y3FRM7UHDHE53JPVDTQUFJ36JV7VAHRTD3NZC6NXWSHFS77ALZJQ, seg_pos={0 0}, upload=198229]: Uploading piece... Aug 16 15:59:08.607803457 (190h33m3.038883588s) [upload=198229]: Next returned nil segment Aug 16 15:59:08.607819728 (190h33m3.038899849s) [upload=198229]: Waiting for error group managing segments... Aug 16 15:59:08.899687194 (190h33m3.330767325s) [noise=true , address=104.36.82.7:4912 , piece_id=UHSAC26VUMU4O5I23DYOHAWZOSAYHAB6DAF7U3W6KNHTIIX7EV7Q, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:08.992904104 (190h33m3.423984235s) [noise=true , address=104.36.82.12:4149 , piece_id=OLBI5RRDUMHYQ7522VEHWDMRAGNKWTEBAZKMJGBY3WBY4SMZDQCA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:11.376957102 (190h33m5.808037233s) [noise=true , address=45.91.116.6:4975 , piece_id=VKSCDFJILOCFYXTB3OFPTTXZBUQ3W74VS7GTUQIVWYVITJC3TCGQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:11.457074829 (190h33m5.88815496s) [noise=true , address=77.90.31.13:4627 , piece_id=PWNUKSQLM54WA3ZMHMWGSG5XGJ67A7QVGC4G5LEQT3TVFY6DCWQA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:11.857435334 (190h33m6.288515465s) [noise=true , address=45.91.116.13:4155 , piece_id=345OQ37OEAQOKMVZN6SFC4UDXPQO57OD2F2DK4WHHQ7KZ55YKUIQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:13.271062370 (190h33m7.702142501s) [noise=true , address=45.91.116.9:4263 , piece_id=Y3FRM7UHDHE53JPVDTQUFJ36JV7VAHRTD3NZC6NXWSHFS77ALZJQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:14.934988376 (190h33m9.366068507s) [noise=true , address=77.90.31.14:4661 , piece_id=OTLHFI2LFB2Z2NYFTJISFHRGQUJ6XHCA32GRM7WF6BZ463543UOQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:18.779363546 (190h33m13.210443667s) [noise=true , address=45.91.116.6:4973 , piece_id=GHAZQGUX7S6NNNJT73QEURUTU43D4PR7MY2DIKHWXNIHFIUU2EWA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:21.390632806 (190h33m15.821712937s) [noise=true , address=45.91.116.14:4189 , piece_id=IYI6IFI5MI5GFVVMTHJV6WCANYULN2C2VEHLC2DE3LUA7QQKD62Q, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:23.053654883 (190h33m17.484735014s) [noise=true , address=45.91.116.5:4297 , piece_id=HXOBR2IKERVXURVJMELFKAJ7IHWJHDNFDTZTOKQIM3TITRBE3CYA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:25.244244521 (190h33m19.675324652s) [noise=true , address=77.90.31.15:4699 , piece_id=VFGEO5AOPAUNWY5KITKM6PZNJXMPPPZDNZ4XLHBLZY3RTPZM6IAA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:27.922866170 (190h33m22.353946301s) [noise=true , address=77.90.31.16:4734 , piece_id=K4AHIZJG4LIAEZMEGHRVLTIQBCZRL666M4N5LRQ2U4URRH2VVE7A, seg_pos={0 0}, upload=198229]: Done uploading piece. err: <nil> Aug 16 15:59:35.744065544 (190h33m30.175145675s) [noise=true , address=104.36.82.13:4185 , piece_id=ZTLWEEWTXK7ES3SP2DXAZHGQVT3APRTE24VAAOYOG5UYAZLWRLFA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:45830->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:35.744085822 (190h33m30.175165953s) [noise=true , address=45.91.116.8:4048 , piece_id=74SZV342HX3KGO7KUJK5HH2TNRB2QHSHUL6EWVMYR3MGKLJ3BSQQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:51758->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:51758->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51758->45.91.116.8:4048: use of closed network connection Aug 16 15:59:35.744111662 (190h33m30.175191783s) [noise=true , address=104.36.82.2:4730 , piece_id=AKGJIR4BVKO2S5BR5BQO7FX5ZTG3LV62C6PC2FKPT3BRTEDPSDSA, seg_pos={0 0}, upload=198229]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:60360->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:35.744150146 (190h33m30.175230267s) [noise=true , address=77.90.31.11:4555 , piece_id=5MM5BLXB53ECYEYFRR7F7PMK4OSUND3C63MEEXB4U72HWISRQJIQ, seg_pos={0 0}, upload=198229]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37984->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:35.744169563 (190h33m30.175249694s) [seg_pos={0 0}, upload=198229]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:59:35.971085392 (190h33m30.402165523s) [upload=198229]: Done uploading segments... Aug 16 15:59:35.971113035 (190h33m30.402193166s) [upload=198229]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.228465582s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 8.435960828s FAIL (2025-08-16 15:59:30 UTC) orbiter warning: context deadline exceeded Duration: 8.435975949s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 21.198312889s FAIL (2025-08-16 15:59:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.198327686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.226980686s FAIL (2025-08-16 15:59:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ng piece... Aug 16 15:58:03.635682156 (190h31m58.066762287s) [noise=true , address=77.90.31.11:4579 , piece_id=XUYCGCOHQWQARFPVPZR6ZXHYFTFXZRS3L2MEKPZW5RY26Y2E5MXQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.636024150 (190h31m58.067104281s) [noise=true , address=79.127.220.101:4073 , piece_id=XIYYPRDJ3K5JDTEPZIY5TQLCEFDLZH5RKSDTWHTL4QNZZNQIS5AA, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.636784906 (190h31m58.067865037s) [noise=true , address=104.36.82.60:4355 , piece_id=MJCTUF2724WLC7MVJ2ONNPRNMYZZFO4SXXLJ5LOBZ2MFQBQTXRSA, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.637037469 (190h31m58.0681176s) [noise=true , address=104.36.82.39:4735 , piece_id=HUGSY54AX4U63GUTQJRXZ653WQ22VXKG6L27USM2SENTD4R6L2AA, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.637320790 (190h31m58.068400921s) [noise=true , address=77.90.31.16:4764 , piece_id=CRYEC4JGLCFPB2J7Q2QHPGZU2AFPZWPDIBBIAR4CXD5KO2DBEG2Q, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.637545019 (190h31m58.06862515s) [noise=true , address=77.90.31.15:4701 , piece_id=OG2T2C4XCMQZ2QCRB72YXV5Y7WROHANWIR6G7BCBG5UO3BBJ6SWA, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.637764469 (190h31m58.0688446s) [noise=true , address=66.23.193.175:4559 , piece_id=E42YNW5SCF3XUYVSVX7RPLMFGE6VZJZGF625KQAHZDS3UQ26YSHQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.638168643 (190h31m58.069248774s) [noise=true , address=104.36.82.5:4864 , piece_id=5CFBTZP74DKR4BHEJVQQDYFDTFRL4Y3PUPXKGLT63CLSDDCHBGSQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.638374287 (190h31m58.069454418s) [noise=true , address=77.90.31.10:4538 , piece_id=GXFNYETMXDTX3JWYPYZNPMPLQOICAMGJG63VVIWRTKBZIZJZWW7Q, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.638605990 (190h31m58.069686111s) [noise=true , address=104.36.82.34:4553 , piece_id=GQ232D5XVYBJO6QSD7PWK4QRJXJ6PQ4754C43HMZO3ATTUBC7NFA, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.638870827 (190h31m58.069950958s) [noise=true , address=104.36.82.23:4459 , piece_id=U4QGM3AV3JR4GCDFTCTFAH72LDI3NG5WHJM5M3OW36A754IGZ5DQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.639145913 (190h31m58.070226044s) [noise=true , address=104.36.82.38:4701 , piece_id=YRMMYMKM7BJQJNY4NVZNN3BUVKZMYUT3NPJI5E5UHU7QN3PSXXJQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.639355273 (190h31m58.070435404s) [noise=true , address=104.36.82.4:4806 , piece_id=IZYI7FU5KBV7TYF45Q2MDUEMDYS5KVUSL4T2E25G3QPZER6PDKSQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:03.639598168 (190h31m58.070678299s) [noise=true , address=104.36.82.12:4162 , piece_id=XY4G5UXVLDIWIYK2IFZUR5RSFIRE6YZVKI337267DN3FP32XFBLQ, seg_pos={0 0}, upload=198211]: Uploading piece... Aug 16 15:58:04.816551365 (190h31m59.247631496s) [upload=198211]: Next returned nil segment Aug 16 15:58:04.816575521 (190h31m59.247655652s) [upload=198211]: Waiting for error group managing segments... Aug 16 15:58:05.487705814 (190h31m59.918785945s) [noise=true , address=79.127.220.101:4073 , piece_id=XIYYPRDJ3K5JDTEPZIY5TQLCEFDLZH5RKSDTWHTL4QNZZNQIS5AA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:05.959889715 (190h32m0.390969846s) [noise=true , address=77.90.31.15:4701 , piece_id=OG2T2C4XCMQZ2QCRB72YXV5Y7WROHANWIR6G7BCBG5UO3BBJ6SWA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:06.244457763 (190h32m0.675537944s) [noise=true , address=77.90.31.19:4869 , piece_id=CX4ORPHZ3ZFNXNF5K4PD7D7JJ5XJJDGO4TFH5VXCNU2UNEXV3FIA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:06.314358851 (190h32m0.745438983s) [noise=true , address=77.90.31.16:4764 , piece_id=CRYEC4JGLCFPB2J7Q2QHPGZU2AFPZWPDIBBIAR4CXD5KO2DBEG2Q, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:06.464326968 (190h32m0.895407099s) [noise=true , address=77.90.31.10:4538 , piece_id=GXFNYETMXDTX3JWYPYZNPMPLQOICAMGJG63VVIWRTKBZIZJZWW7Q, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:13.392041080 (190h32m7.823121211s) [noise=true , address=66.23.193.175:4559 , piece_id=E42YNW5SCF3XUYVSVX7RPLMFGE6VZJZGF625KQAHZDS3UQ26YSHQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:58:25.595571935 (190h32m20.026652066s) [noise=true , address=104.36.82.12:4162 , piece_id=XY4G5UXVLDIWIYK2IFZUR5RSFIRE6YZVKI337267DN3FP32XFBLQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: <nil> Aug 16 15:59:02.527882109 (190h32m56.95896223s) [noise=true , address=104.36.82.44:4898 , piece_id=UZLYJSISQLXRFVPU4D53BKGXL6H6RLD4AIE722VSDGTW47MKSP7Q, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: write tcp 188.166.225.170:42628->104.36.82.44:4898: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.527956510 (190h32m56.959036641s) [noise=true , address=104.36.82.38:4701 , piece_id=YRMMYMKM7BJQJNY4NVZNN3BUVKZMYUT3NPJI5E5UHU7QN3PSXXJQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:1weCZWAtZRRYdeuPE8TjnZB5fZVD4gz4qW6TxyyNz33wVfW9hQ, address:104.36.82.38:4701): protocol: rpc: dial tcp 104.36.82.38:4701: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.527993221 (190h32m56.959073352s) [noise=true , address=104.36.82.60:4355 , piece_id=MJCTUF2724WLC7MVJ2ONNPRNMYZZFO4SXXLJ5LOBZ2MFQBQTXRSA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12eA7ZLTkgGyCRsnQSEC8oYM3y9oeB5MQ4FUueJNMb4FLZueNv2, address:104.36.82.60:4355): protocol: rpc: dial tcp 104.36.82.60:4355: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528034089 (190h32m56.95911422s) [noise=true , address=104.36.82.23:4459 , piece_id=U4QGM3AV3JR4GCDFTCTFAH72LDI3NG5WHJM5M3OW36A754IGZ5DQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528056873 (190h32m56.959137004s) [noise=true , address=104.36.82.34:4553 , piece_id=GQ232D5XVYBJO6QSD7PWK4QRJXJ6PQ4754C43HMZO3ATTUBC7NFA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:123VKpEDNfLtYfypf6MD4TsSm7Svho8cdB7iD4HAqefNtC1WgcR, address:104.36.82.34:4553): protocol: rpc: dial tcp 104.36.82.34:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528080849 (190h32m56.95916098s) [noise=true , address=104.36.82.5:4864 , piece_id=5CFBTZP74DKR4BHEJVQQDYFDTFRL4Y3PUPXKGLT63CLSDDCHBGSQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: rpc: dial tcp 104.36.82.5:4864: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528104023 (190h32m56.959184154s) [noise=true , address=104.36.82.58:4276 , piece_id=LODCYKD32OO7KY56R3TMZGITLOALU2B467TXY76HNF2FS44UL7NQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:126tQjCVpoLF4zbVbNH37ZTVbRRt2EK3zndnitxZs5xeQj2CzWr, address:104.36.82.58:4276): protocol: rpc: dial tcp 104.36.82.58:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528117328 (190h32m56.959197459s) [noise=true , address=104.36.82.4:4806 , piece_id=IZYI7FU5KBV7TYF45Q2MDUEMDYS5KVUSL4T2E25G3QPZER6PDKSQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12qUyweKQhw15RjewCgKi1oLyCvN3VUB1VMpeGhj2Q2hszvJBsQ, address:104.36.82.4:4806): protocol: rpc: dial tcp 104.36.82.4:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528159960 (190h32m56.959240091s) [noise=true , address=104.36.82.35:4604 , piece_id=75VGFXMY6YSBRSSD3KNY555FGJQE7YHVV75TV6S3JY72WPZSXJBA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12EhoYFXjEa1ciTS2UXFRFUWhfSyd7p3oPDgqrdNVTYyBUk4mJ1, address:104.36.82.35:4604): protocol: write tcp 188.166.225.170:40722->104.36.82.35:4604: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528166092 (190h32m56.959246223s) [noise=true , address=104.36.82.39:4735 , piece_id=HUGSY54AX4U63GUTQJRXZ653WQ22VXKG6L27USM2SENTD4R6L2AA, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:1QCQZ8eFGLa5TAaAJJzUip8o5DAELzTBTm4UFbt3Am5yYWXAZF, address:104.36.82.39:4735): protocol: rpc: dial tcp 104.36.82.39:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528173836 (190h32m56.959253967s) [noise=true , address=77.90.31.11:4579 , piece_id=XUYCGCOHQWQARFPVPZR6ZXHYFTFXZRS3L2MEKPZW5RY26Y2E5MXQ, seg_pos={0 0}, upload=198211]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: rpc: dial tcp 77.90.31.11:4579: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:59:02.528187262 (190h32m56.959267393s) [seg_pos={0 0}, upload=198211]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:59:02.753082234 (190h32m57.184162365s) [upload=198211]: Done uploading segments... Aug 16 15:59:02.753110057 (190h32m57.184190188s) [upload=198211]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.226995004s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 19.544353055s FAIL (2025-08-16 15:58:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.544375368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233767597s FAIL (2025-08-16 15:58:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Aug 16 15:57:36.834229898 (190h31m31.265310029s) [upload=198204]: Starting upload Aug 16 15:57:36.834306405 (190h31m31.265386536s) [upload=198204]: Done starting upload Aug 16 15:57:36.834384244 (190h31m31.265464375s) [upload=198204]: Uploading segments... Aug 16 15:57:36.834599896 (190h31m31.265680027s) [upload=198204]: Got next segment. Inline: false Aug 16 15:57:36.834633591 (190h31m31.265713722s) [upload=198204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:57:36.834638219 (190h31m31.26571835s) [upload=198204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:57:37.346911767 (190h31m31.777991898s) [seg_pos={0 0}, upload=198204]: Begin upload segment... Aug 16 15:57:37.346951964 (190h31m31.778032095s) [seg_pos={0 0}, upload=198204]: Done begin upload segment. Aug 16 15:57:37.346974527 (190h31m31.778054648s) [noise=true , address=66.23.193.175:4555 , piece_id=5NNH22KHXQUW2PH3MPYVINQKA2ZFF7PB5NUD63UT4ZC7IFPLBNUQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.347014103 (190h31m31.778094234s) [noise=true , address=45.91.116.5:4324 , piece_id=XM43XHWX53DA4JCNRS7PRR6FAQL6XSEZSUYZQBULPC6UNQISA6UQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.347339686 (190h31m31.778419817s) [noise=true , address=104.36.82.5:4841 , piece_id=6M4EY36DQFZYJGZA2TFFFCAJ3LSTRWYVMSBKUQU3AZBTUY6TQX7A, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.347482218 (190h31m31.778562349s) [noise=true , address=77.90.31.18:4829 , piece_id=73MD2JBSJPC53V6NSAUXIFVTGCMHAVNZJ4RMG4R5XWCFZJB26A6Q, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.347662774 (190h31m31.778742905s) [noise=true , address=104.36.82.39:4737 , piece_id=LC5ND4MRZGJUVPUDI3TFFX3F7ORKRG5FJLTKFZY7TVYBF6WEZZ5Q, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.347958180 (190h31m31.779038311s) [noise=true , address=77.90.31.21:4906 , piece_id=Z2PYO2SAIROPEPATBHTAERRR55XV264OQY4XUHLEU7BGKE3O5A7Q, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.348248585 (190h31m31.779328716s) [noise=true , address=104.36.82.58:4266 , piece_id=AC2PKY3JD7OENHGTLF5O6BGJI4YUDAB2DRE5YMXHSKQGOPLFWMWQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.348449450 (190h31m31.779529581s) [noise=true , address=79.127.201.215:4351 , piece_id=QBV3LLUDSVLNGYG255K3LY2PLIQSA3V3YO7CR4EI2T7A575ZPLEQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.348755225 (190h31m31.779835346s) [noise=true , address=104.36.82.36:4634 , piece_id=JQEPIOEAAWDWXW63R7M4GVT2FBFMEHU6JFEP44H47ZOA2ZEETCOA, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.349052403 (190h31m31.780132534s) [noise=true , address=77.90.31.10:4542 , piece_id=KCZAX545OB4OZYUOSEV2VIXBIANI66RQ5SCPPW4IBXH3BXGLB3OA, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.349223641 (190h31m31.780303762s) [noise=true , address=104.36.82.60:4361 , piece_id=JBP4FSZCSVV5SMPVDTNVFYGN67OT3QE6I4UNHBNFZBEGHNIV2AXQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.349477296 (190h31m31.780557427s) [noise=true , address=77.90.31.19:4846 , piece_id=246HUF5BZJIBDT5X2SOCYNR36QVEMW6LY32X5PAJZ2DRAGBL54AQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.349722335 (190h31m31.780802466s) [noise=true , address=104.36.82.34:4576 , piece_id=3N3V6A6CUP6TCIJO64RX2BP6PIHFIOED3OLLD3YAICCJCISPM3WA, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.350195991 (190h31m31.781276122s) [noise=true , address=104.36.82.38:4719 , piece_id=K7AZOGH2S4WRI2V7HZQDKBRS4CW5APZOKHJFISGEJ4CIFXBUEVJQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.350858468 (190h31m31.781938599s) [noise=true , address=77.90.31.11:4581 , piece_id=Z75FJRSJ5QNEDA43GER4UTL4OLNHPKBZ2HVKI5U7RKC2FCQ5FW4A, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.353879489 (190h31m31.784959611s) [noise=true , address=104.36.82.23:4462 , piece_id=A726G4FFS7EOABVG35C2AS2YUCLMVHKUHCYFUU3H3QQYMP4OH5SQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.354434431 (190h31m31.785514562s) [noise=true , address=104.36.82.3:4799 , piece_id=XKDCZ2237KVCOIL2JG2MNH25PZP5RWWVOPCXBL7N2OPA4E65SYYA, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:37.355061551 (190h31m31.786141682s) [noise=true , address=77.90.31.15:4732 , piece_id=MVLWVRA4YLHQLA25JSPAUAYZNE6MAADNNO5BPAORNBRFFMRWWNWQ, seg_pos={0 0}, upload=198204]: Uploading piece... Aug 16 15:57:38.480074427 (190h31m32.911154558s) [upload=198204]: Next returned nil segment Aug 16 15:57:38.480091268 (190h31m32.911171389s) [upload=198204]: Waiting for error group managing segments... Aug 16 15:57:39.472060082 (190h31m33.903140213s) [noise=true , address=77.90.31.18:4829 , piece_id=73MD2JBSJPC53V6NSAUXIFVTGCMHAVNZJ4RMG4R5XWCFZJB26A6Q, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:39.573302678 (190h31m34.004382809s) [noise=true , address=77.90.31.21:4906 , piece_id=Z2PYO2SAIROPEPATBHTAERRR55XV264OQY4XUHLEU7BGKE3O5A7Q, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:39.670833562 (190h31m34.101913723s) [noise=true , address=77.90.31.19:4846 , piece_id=246HUF5BZJIBDT5X2SOCYNR36QVEMW6LY32X5PAJZ2DRAGBL54AQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:43.805252808 (190h31m38.236332939s) [noise=true , address=79.127.201.215:4351 , piece_id=QBV3LLUDSVLNGYG255K3LY2PLIQSA3V3YO7CR4EI2T7A575ZPLEQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:44.338965257 (190h31m38.770045388s) [noise=true , address=45.91.116.5:4324 , piece_id=XM43XHWX53DA4JCNRS7PRR6FAQL6XSEZSUYZQBULPC6UNQISA6UQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:47.164117216 (190h31m41.595197347s) [noise=true , address=66.23.193.175:4555 , piece_id=5NNH22KHXQUW2PH3MPYVINQKA2ZFF7PB5NUD63UT4ZC7IFPLBNUQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:52.323771535 (190h31m46.754851696s) [noise=true , address=77.90.31.15:4732 , piece_id=MVLWVRA4YLHQLA25JSPAUAYZNE6MAADNNO5BPAORNBRFFMRWWNWQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:57:55.461873710 (190h31m49.892953841s) [noise=true , address=77.90.31.10:4542 , piece_id=KCZAX545OB4OZYUOSEV2VIXBIANI66RQ5SCPPW4IBXH3BXGLB3OA, seg_pos={0 0}, upload=198204]: Done uploading piece. err: <nil> Aug 16 15:58:36.176320070 (190h32m30.607400201s) [noise=true , address=104.36.82.23:4462 , piece_id=A726G4FFS7EOABVG35C2AS2YUCLMVHKUHCYFUU3H3QQYMP4OH5SQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: write tcp 188.166.225.170:42916->104.36.82.23:4462: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.176911040 (190h32m30.607991161s) [noise=true , address=104.36.82.38:4719 , piece_id=K7AZOGH2S4WRI2V7HZQDKBRS4CW5APZOKHJFISGEJ4CIFXBUEVJQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:12r5RXjrbpRcyxvcYhpoqQdgv8MD8zLmpninS7D3cdNCid4Rc8j, address:104.36.82.38:4719): protocol: rpc: dial tcp 104.36.82.38:4719: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.176952059 (190h32m30.60803219s) [noise=true , address=104.36.82.34:4576 , piece_id=3N3V6A6CUP6TCIJO64RX2BP6PIHFIOED3OLLD3YAICCJCISPM3WA, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus, address:104.36.82.34:4576): protocol: rpc: dial tcp 104.36.82.34:4576: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.176986525 (190h32m30.608066656s) [noise=true , address=104.36.82.39:4737 , piece_id=LC5ND4MRZGJUVPUDI3TFFX3F7ORKRG5FJLTKFZY7TVYBF6WEZZ5Q, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:12mQUrPb7pxaBQgFR8rZEnzvDSabb9RrejsY3NWjDTEZY6X2A8t, address:104.36.82.39:4737): protocol: rpc: dial tcp 104.36.82.39:4737: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177009609 (190h32m30.60808974s) [noise=true , address=77.90.31.11:4581 , piece_id=Z75FJRSJ5QNEDA43GER4UTL4OLNHPKBZ2HVKI5U7RKC2FCQ5FW4A, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: rpc: dial tcp 77.90.31.11:4581: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177022453 (190h32m30.608102584s) [noise=true , address=104.36.82.58:4266 , piece_id=AC2PKY3JD7OENHGTLF5O6BGJI4YUDAB2DRE5YMXHSKQGOPLFWMWQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:104.36.82.58:4266): protocol: rpc: dial tcp 104.36.82.58:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177030469 (190h32m30.60811059s) [noise=true , address=104.36.82.60:4361 , piece_id=JBP4FSZCSVV5SMPVDTNVFYGN67OT3QE6I4UNHBNFZBEGHNIV2AXQ, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: rpc: dial tcp 104.36.82.60:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177058723 (190h32m30.608138844s) [noise=true , address=104.36.82.3:4799 , piece_id=XKDCZ2237KVCOIL2JG2MNH25PZP5RWWVOPCXBL7N2OPA4E65SYYA, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: rpc: dial tcp 104.36.82.3:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177102867 (190h32m30.608182998s) [noise=true , address=104.36.82.5:4841 , piece_id=6M4EY36DQFZYJGZA2TFFFCAJ3LSTRWYVMSBKUQU3AZBTUY6TQX7A, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: write tcp 188.166.225.170:57518->104.36.82.5:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177142554 (190h32m30.608222685s) [noise=true , address=104.36.82.36:4634 , piece_id=JQEPIOEAAWDWXW63R7M4GVT2FBFMEHU6JFEP44H47ZOA2ZEETCOA, seg_pos={0 0}, upload=198204]: Done uploading piece. err: ecclient: upload failed (node:1kwydp3q2wTrCVadJZB7WmKGK8h2R5QL7HGpUsjTpRbcnWUSb5, address:104.36.82.36:4634): protocol: write tcp 188.166.225.170:42660->104.36.82.36:4634: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:36.177158393 (190h32m30.608238524s) [seg_pos={0 0}, upload=198204]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:58:36.408526059 (190h32m30.839606201s) [upload=198204]: Done uploading segments... Aug 16 15:58:36.408557340 (190h32m30.839637471s) [upload=198204]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.233782836s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........X.X..X.....X......................X................] duration: 2.029336247s FAIL (2025-08-16 15:58:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.029363761s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [................X......X..XXXXXXXXXX.XX.XXXXXX......X.......] duration: 7.022896325s FAIL (2025-08-16 15:58:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.022921443s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.41896663s FAIL (2025-08-16 15:58:25 UTC) orbiter warning: context deadline exceeded Duration: 6.418990507s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 22.495721677s FAIL (2025-08-16 15:58:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.495746205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228837678s FAIL (2025-08-16 15:58:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== QRICZCWZWE43JCXUNT2MQ4WAQZPXTIRUNTOLCYCPOAJXKLLAKCXQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.183521769 (190h30m57.6146019s) [noise=true , address=104.36.82.60:4358 , piece_id=OMP2V6QNICXJF5F2LVBJRVG4BD2PPT3YA34VJVW635XOKA2GI74Q, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.183802526 (190h30m57.614882658s) [noise=true , address=104.36.82.34:4572 , piece_id=6PI33CKZYLMI2KA7Y4YTYON63LE2XXBR77RKUJNZEGWE3EDLNZRQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.184073534 (190h30m57.615153665s) [noise=true , address=79.127.201.215:4344 , piece_id=DIZWVXX52WFYEN5UQBHRTAFLQKNGAS2O2H3GO44RH74KM3CTL2IA, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.184370122 (190h30m57.615450253s) [noise=true , address=104.36.82.39:4730 , piece_id=T2TFZ5WB7EFUJLJFI55ET7YUJUZSXSLKWRMA64WIMTWK5K7ZHE5A, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.184614189 (190h30m57.61569431s) [noise=true , address=104.36.82.35:4601 , piece_id=77QROFBNH4J4AY2RMG6NHH7EZ233II7DSKGXNHNN7MMG2SGOL6UA, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.184839740 (190h30m57.615919871s) [noise=true , address=77.90.31.13:4629 , piece_id=WYUWT63TGIGIQXRMSN77IGBYZ5FIJHOL72PDZ7GPGETOASCKBHQQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.185066976 (190h30m57.616147107s) [noise=true , address=79.127.220.101:4060 , piece_id=2IH5O3F7BK473WO3ZMATDEOCDJQZVKXS32ICWB4X57ELNE2IQHAQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.185307896 (190h30m57.616388027s) [noise=true , address=77.90.31.11:4581 , piece_id=EPW2LQONLKPKRU67ZGIBREUAF2MRSGJNB2KXFUIIBYNHBUPDKETQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.185509312 (190h30m57.616589433s) [noise=true , address=45.91.116.7:4016 , piece_id=ZPSTELUGYB2GJWF4BMDT5QMO35YALHHI6J75RSI63UWKJPKSX76A, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.185713462 (190h30m57.616793593s) [noise=true , address=104.36.82.58:4280 , piece_id=4SZY2TVWCFB4ZANVTO5T3UQWFVLURP273MQVN3PHJG4MZYHM77FQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.185948913 (190h30m57.617029044s) [noise=true , address=104.36.82.9:4025 , piece_id=DTI5JAKNWSZFAHZNROSEZWY7IHH3JPZKDJHF5II4CW5Z3PTFIMSQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.186182630 (190h30m57.617262761s) [noise=true , address=77.90.31.15:4704 , piece_id=WR7HSJZKDHPC27AFQZKRORVPOM6UPWFQLPIPXJMTAKMOAOO4FJXQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.186397671 (190h30m57.617477802s) [noise=true , address=104.36.82.23:4459 , piece_id=TK7HGYR4EESOVRIUMPRJANAICRBZMN5FXSLFIRVAVWIMW23HJR2Q, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.186599869 (190h30m57.61767999s) [noise=true , address=45.91.116.8:4072 , piece_id=U4YW3SWDWJ7PVO5E2VF4X3RWMI7I7EMR4W6T5L3HRCO3TGQW6TOQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.186833255 (190h30m57.617913386s) [noise=true , address=77.90.31.19:4854 , piece_id=OGJPJYBFCN33TW4X3HY6QVQQMG74ZASRHEFYR23Z7YTLMFRFSMKA, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:03.187024561 (190h30m57.618104692s) [noise=true , address=45.91.116.12:4257 , piece_id=G2JDL53VDJUULSEAMOGT6WYUHSX2QDJJJWWNIIOFXPDADSOJRPJQ, seg_pos={0 0}, upload=198197]: Uploading piece... Aug 16 15:57:04.680255351 (190h30m59.111335472s) [upload=198197]: Next returned nil segment Aug 16 15:57:04.680310557 (190h30m59.111390688s) [upload=198197]: Waiting for error group managing segments... Aug 16 15:57:05.371525469 (190h30m59.8026056s) [noise=true , address=104.36.82.9:4025 , piece_id=DTI5JAKNWSZFAHZNROSEZWY7IHH3JPZKDJHF5II4CW5Z3PTFIMSQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:05.403255471 (190h30m59.834335602s) [noise=true , address=77.90.31.14:4696 , piece_id=4PERRFQ72LBUB7JX5VIJDE7TRRL75Q6PNIRHRVXM65RMPDCDU4LQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:05.419178138 (190h30m59.850258269s) [noise=true , address=45.91.116.7:4016 , piece_id=ZPSTELUGYB2GJWF4BMDT5QMO35YALHHI6J75RSI63UWKJPKSX76A, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:05.431239495 (190h30m59.862319626s) [noise=true , address=77.90.31.13:4629 , piece_id=WYUWT63TGIGIQXRMSN77IGBYZ5FIJHOL72PDZ7GPGETOASCKBHQQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:05.621310633 (190h31m0.052390764s) [noise=true , address=45.91.116.12:4257 , piece_id=G2JDL53VDJUULSEAMOGT6WYUHSX2QDJJJWWNIIOFXPDADSOJRPJQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:05.720329446 (190h31m0.151409577s) [noise=true , address=77.90.31.19:4854 , piece_id=OGJPJYBFCN33TW4X3HY6QVQQMG74ZASRHEFYR23Z7YTLMFRFSMKA, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:06.461340998 (190h31m0.892421129s) [noise=true , address=79.127.220.101:4060 , piece_id=2IH5O3F7BK473WO3ZMATDEOCDJQZVKXS32ICWB4X57ELNE2IQHAQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:08.877848825 (190h31m3.308928956s) [noise=true , address=45.91.116.8:4072 , piece_id=U4YW3SWDWJ7PVO5E2VF4X3RWMI7I7EMR4W6T5L3HRCO3TGQW6TOQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:57:10.001762977 (190h31m4.432843118s) [noise=true , address=79.127.201.215:4344 , piece_id=DIZWVXX52WFYEN5UQBHRTAFLQKNGAS2O2H3GO44RH74KM3CTL2IA, seg_pos={0 0}, upload=198197]: Done uploading piece. err: <nil> Aug 16 15:58:02.073090350 (190h31m56.504170482s) [noise=true , address=77.90.31.11:4581 , piece_id=EPW2LQONLKPKRU67ZGIBREUAF2MRSGJNB2KXFUIIBYNHBUPDKETQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:58:02.073321393 (190h31m56.504401534s) [noise=true , address=104.36.82.34:4572 , piece_id=6PI33CKZYLMI2KA7Y4YTYON63LE2XXBR77RKUJNZEGWE3EDLNZRQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:124jj7LpTJR3kta3gK57SzNwATM9NuKpNCwkiC2zzedeH7YBAf3, address:104.36.82.34:4572): protocol: rpc: dial tcp 104.36.82.34:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073390966 (190h31m56.504471097s) [noise=true , address=104.36.82.58:4280 , piece_id=4SZY2TVWCFB4ZANVTO5T3UQWFVLURP273MQVN3PHJG4MZYHM77FQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1fgHaEFKK4CeGg4x3vywTuYfT3kmac31xqg2QpsahQNXmvjhW8, address:104.36.82.58:4280): protocol: write tcp 188.166.225.170:54978->104.36.82.58:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073412838 (190h31m56.504492979s) [noise=true , address=77.90.31.15:4704 , piece_id=WR7HSJZKDHPC27AFQZKRORVPOM6UPWFQLPIPXJMTAKMOAOO4FJXQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: write tcp 188.166.225.170:55472->77.90.31.15:4704: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073438617 (190h31m56.504518758s) [noise=true , address=104.36.82.23:4459 , piece_id=TK7HGYR4EESOVRIUMPRJANAICRBZMN5FXSLFIRVAVWIMW23HJR2Q, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1hGJmw1LEAdN7C4j5ScEfhugkFneArUxcAuneCMGwsKNjQpn2N, address:104.36.82.23:4459): protocol: rpc: dial tcp 104.36.82.23:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073492931 (190h31m56.504573062s) [noise=true , address=104.36.82.5:4851 , piece_id=QRICZCWZWE43JCXUNT2MQ4WAQZPXTIRUNTOLCYCPOAJXKLLAKCXQ, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1fTtTieEUquaKNUNzqK98PkX7tVxjsiNeMLjz8kQJwdLne7d9R, address:104.36.82.5:4851): protocol: rpc: dial tcp 104.36.82.5:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073506767 (190h31m56.504586898s) [noise=true , address=104.36.82.35:4601 , piece_id=77QROFBNH4J4AY2RMG6NHH7EZ233II7DSKGXNHNN7MMG2SGOL6UA, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:16ggANPGWDLtyxp9hMkCsNYueB2KuVWsKVrG8mFa6pPBXW5RiW, address:104.36.82.35:4601): protocol: rpc: dial tcp 104.36.82.35:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073546343 (190h31m56.504626484s) [noise=true , address=104.36.82.60:4358 , piece_id=OMP2V6QNICXJF5F2LVBJRVG4BD2PPT3YA34VJVW635XOKA2GI74Q, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1dwCNEcA9o51ZN3UaN3Frxdawwf8DXBj86iQMywjW8XP4ESmw1, address:104.36.82.60:4358): protocol: rpc: dial tcp 104.36.82.60:4358: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073556512 (190h31m56.504636643s) [noise=true , address=104.36.82.39:4730 , piece_id=T2TFZ5WB7EFUJLJFI55ET7YUJUZSXSLKWRMA64WIMTWK5K7ZHE5A, seg_pos={0 0}, upload=198197]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:104.36.82.39:4730): protocol: rpc: dial tcp 104.36.82.39:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:58:02.073582723 (190h31m56.504662854s) [seg_pos={0 0}, upload=198197]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:58:02.300178118 (190h31m56.731258249s) [upload=198197]: Done uploading segments... Aug 16 15:58:02.300200380 (190h31m56.731280511s) [upload=198197]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.228867927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........X.X..X.....X......................X...............] duration: 2.018326431s FAIL (2025-08-16 15:57:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.018353793s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [..X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.322187043s FAIL (2025-08-16 15:57:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 16 15:56:36.264630770 (190h30m30.695710901s) [upload=198189]: Starting upload Aug 16 15:56:36.264674063 (190h30m30.695754194s) [upload=198189]: Done starting upload Aug 16 15:56:36.264799062 (190h30m30.695879213s) [upload=198189]: Uploading segments... Aug 16 15:56:36.264893052 (190h30m30.695973183s) [upload=198189]: Got next segment. Inline: false Aug 16 15:56:36.264923300 (190h30m30.696003431s) [upload=198189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:36.264927769 (190h30m30.6960079s) [upload=198189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:36.838130368 (190h30m31.269210499s) [seg_pos={0 0}, upload=198189]: Begin upload segment... Aug 16 15:56:36.838171427 (190h30m31.269251548s) [seg_pos={0 0}, upload=198189]: Done begin upload segment. Aug 16 15:56:36.838194040 (190h30m31.269274161s) [noise=true , address=104.36.82.60:4339 , piece_id=FNM45JW56FUKYH4AZP6VUOLV3ZOJ2XRA3CC3D225JKUMNI66PHXA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.838207115 (190h30m31.269287246s) [noise=true , address=104.36.82.59:4335 , piece_id=MPGSHYNA2A32EHMVES626OPVVCQQCPQFXTWH6SX2RHBEFNGCWKXQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.838543959 (190h30m31.26962409s) [noise=true , address=45.91.116.7:4016 , piece_id=D5CTYIJ4YYPLNMNWAVORMTLP7UGTATAYPYJTM7CSIOO2OWJVTWWQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.838553107 (190h30m31.269633228s) [noise=true , address=66.23.193.175:4557 , piece_id=YTX7TV67BAD5GI7AJYFDY36P6OZBK4TSRCOJG24LKDG2MEQU6EGA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.838838042 (190h30m31.269918163s) [noise=true , address=77.90.31.16:4760 , piece_id=QH6O2462YZVXXF3KOGARHKJQKLXQHUWGVHOTGE3QHUKEER7EZIIQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.838852370 (190h30m31.269932491s) [noise=true , address=104.36.82.58:4272 , piece_id=VPA6SHJWS76MEBEU3RXXNGAEODP43TSANRN7JWL2PJ4CZF7NMLRQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839103841 (190h30m31.270183972s) [noise=true , address=156.146.43.225:4457 , piece_id=Y7674LLR5MADYTHYMEORRJRZOA2ABO2GN4OBE6N3ENV6CVNSJIJA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839122646 (190h30m31.270202777s) [noise=true , address=104.36.82.34:4565 , piece_id=ZAMCJ35BOUVCO6VXZSHVJLYYFQJVYW3VUIJEFZ4J27GCVBAIVSBA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839381111 (190h30m31.270461242s) [noise=true , address=77.90.31.11:4587 , piece_id=5CC4V2SECWMX2SPKTDWKQNWFDBDAMUGW2VOAGU6II56PPURDVZDQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839447197 (190h30m31.270527328s) [noise=true , address=45.91.116.14:4204 , piece_id=WVQHWF6YR4ZSP5S3UJTKX44TZDM6W6MLNTPBG2OSRKXNYWWQ3FRA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839654003 (190h30m31.270734124s) [noise=true , address=79.127.201.215:4358 , piece_id=GXPCFVFYYMFSRXUEI75XPKYD6AMR4IL2NMPWEPC3HDU6NUD3VQNQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839711964 (190h30m31.270792085s) [noise=true , address=104.36.82.36:4643 , piece_id=F7LBMLAC7BPQDXXOSEJBFC5WFL3OGEVBAX7ONIOFEWQPXPUHUCSA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839855268 (190h30m31.270935399s) [noise=true , address=104.36.82.23:4470 , piece_id=SLB7HSVWNIFWHSDHRT7DUXILVCLDSCHBCJ23I3AQWXRBMFMLFKWA, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.839935963 (190h30m31.271016094s) [noise=true , address=104.36.82.3:4788 , piece_id=CN5P4PHRT6IW5MXGC2MZEXGD5T2BWEN72F4UVQUBKJDVP5JRWLFQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.840164990 (190h30m31.271245122s) [noise=true , address=79.127.205.245:4223 , piece_id=OZUKRZIXHJFYXKLVKRBWJXONW6WEXVUSWGJDKNRPMWB6OTW3AARQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.840401323 (190h30m31.271481444s) [noise=true , address=77.90.31.18:4838 , piece_id=PE4XPBFLSQFSWGYOB73WTPV7UKDU22BXO7T2ANPXFQC4S5JWAU4A, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.840614892 (190h30m31.271695023s) [noise=true , address=45.91.116.10:4107 , piece_id=PGIMRGVGSUHGY2BBWV5LWVKZJD232DEVB24CLVOKWI7FS66YBDFQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:36.841000049 (190h30m31.27208018s) [noise=true , address=77.90.31.12:4597 , piece_id=2MABHAZ2VEY3NMBCZGHASHIHUCEY3XK4P6FEAT33DLSRGFGORIBQ, seg_pos={0 0}, upload=198189]: Uploading piece... Aug 16 15:56:37.976821135 (190h30m32.407901256s) [upload=198189]: Next returned nil segment Aug 16 15:56:37.976838457 (190h30m32.407918588s) [upload=198189]: Waiting for error group managing segments... Aug 16 15:56:38.269850633 (190h30m32.700930764s) [noise=true , address=79.127.205.245:4223 , piece_id=OZUKRZIXHJFYXKLVKRBWJXONW6WEXVUSWGJDKNRPMWB6OTW3AARQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:38.970305014 (190h30m33.401385145s) [noise=true , address=77.90.31.18:4838 , piece_id=PE4XPBFLSQFSWGYOB73WTPV7UKDU22BXO7T2ANPXFQC4S5JWAU4A, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:39.146296615 (190h30m33.577376746s) [noise=true , address=77.90.31.16:4760 , piece_id=QH6O2462YZVXXF3KOGARHKJQKLXQHUWGVHOTGE3QHUKEER7EZIIQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:41.510209627 (190h30m35.941289748s) [noise=true , address=156.146.43.225:4457 , piece_id=Y7674LLR5MADYTHYMEORRJRZOA2ABO2GN4OBE6N3ENV6CVNSJIJA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:42.564710285 (190h30m36.995790416s) [noise=true , address=45.91.116.14:4204 , piece_id=WVQHWF6YR4ZSP5S3UJTKX44TZDM6W6MLNTPBG2OSRKXNYWWQ3FRA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:42.617328555 (190h30m37.048408686s) [noise=true , address=45.91.116.10:4107 , piece_id=PGIMRGVGSUHGY2BBWV5LWVKZJD232DEVB24CLVOKWI7FS66YBDFQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:42.763505588 (190h30m37.194585709s) [noise=true , address=45.91.116.7:4016 , piece_id=D5CTYIJ4YYPLNMNWAVORMTLP7UGTATAYPYJTM7CSIOO2OWJVTWWQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:43.098587860 (190h30m37.529667981s) [noise=true , address=79.127.201.215:4358 , piece_id=GXPCFVFYYMFSRXUEI75XPKYD6AMR4IL2NMPWEPC3HDU6NUD3VQNQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:47.422968226 (190h30m41.854048367s) [noise=true , address=77.90.31.11:4587 , piece_id=5CC4V2SECWMX2SPKTDWKQNWFDBDAMUGW2VOAGU6II56PPURDVZDQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:56:53.318643111 (190h30m47.749723232s) [noise=true , address=66.23.193.175:4557 , piece_id=YTX7TV67BAD5GI7AJYFDY36P6OZBK4TSRCOJG24LKDG2MEQU6EGA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: <nil> Aug 16 15:57:35.616979066 (190h31m30.048059197s) [noise=true , address=104.36.82.23:4470 , piece_id=SLB7HSVWNIFWHSDHRT7DUXILVCLDSCHBCJ23I3AQWXRBMFMLFKWA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:1FT9cn7mxuyJ7xBeCJgyTcJKFRmHEqboqeGWLvygNceEo4LQVF, address:104.36.82.23:4470): protocol: rpc: dial tcp 104.36.82.23:4470: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617026215 (190h31m30.048106347s) [noise=true , address=104.36.82.3:4788 , piece_id=CN5P4PHRT6IW5MXGC2MZEXGD5T2BWEN72F4UVQUBKJDVP5JRWLFQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: rpc: dial tcp 104.36.82.3:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617055572 (190h31m30.048135703s) [noise=true , address=77.90.31.12:4597 , piece_id=2MABHAZ2VEY3NMBCZGHASHIHUCEY3XK4P6FEAT33DLSRGFGORIBQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:127QMaS39fDFZixRP6gVMQkZMefSxLo3ToPctZuDmaARY6nB3H9, address:77.90.31.12:4597): protocol: rpc: dial tcp 77.90.31.12:4597: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617087212 (190h31m30.048167333s) [noise=true , address=104.36.82.58:4272 , piece_id=VPA6SHJWS76MEBEU3RXXNGAEODP43TSANRN7JWL2PJ4CZF7NMLRQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:1yY39UpswsTLoWz4353JykK35fg8KB4AJWPatKGFgqTt4gDMcR, address:104.36.82.58:4272): protocol: rpc: dial tcp 104.36.82.58:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617115296 (190h31m30.048195427s) [noise=true , address=104.36.82.36:4643 , piece_id=F7LBMLAC7BPQDXXOSEJBFC5WFL3OGEVBAX7ONIOFEWQPXPUHUCSA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw, address:104.36.82.36:4643): protocol: rpc: dial tcp 104.36.82.36:4643: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617143891 (190h31m30.048224022s) [noise=true , address=104.36.82.60:4339 , piece_id=FNM45JW56FUKYH4AZP6VUOLV3ZOJ2XRA3CC3D225JKUMNI66PHXA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617173578 (190h31m30.048253699s) [noise=true , address=104.36.82.59:4335 , piece_id=MPGSHYNA2A32EHMVES626OPVVCQQCPQFXTWH6SX2RHBEFNGCWKXQ, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:12gFW6PBz447rjBeSsnrdskgBcCHrqvB1SDCcLBzHU8G972fUm2, address:104.36.82.59:4335): protocol: rpc: dial tcp 104.36.82.59:4335: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617204818 (190h31m30.048284949s) [noise=true , address=104.36.82.34:4565 , piece_id=ZAMCJ35BOUVCO6VXZSHVJLYYFQJVYW3VUIJEFZ4J27GCVBAIVSBA, seg_pos={0 0}, upload=198189]: Done uploading piece. err: ecclient: upload failed (node:1JRxcRCxwuNJrRRcKKxEggFpmRk2fo9kdVbmhPn3eLLeiQop9R, address:104.36.82.34:4565): protocol: write tcp 188.166.225.170:54950->104.36.82.34:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:57:35.617227611 (190h31m30.048307773s) [seg_pos={0 0}, upload=198189]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:57:35.936124634 (190h31m30.367204765s) [upload=198189]: Done uploading segments... Aug 16 15:57:35.936146176 (190h31m30.367226307s) [upload=198189]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.322198465s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 29.232527546s FAIL (2025-08-16 15:57:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.232549018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.................X......X..XXXXXXXXXX.XX.XXXXXX......X......] duration: 7.1665893s FAIL (2025-08-16 15:57:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.166617453s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.461401347s FAIL (2025-08-16 15:57:33 UTC) orbiter warning: context deadline exceeded Duration: 6.461422397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...........X.X..X.....X......................X..............] duration: 2.453095001s FAIL (2025-08-16 15:57:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.453119238s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.378411812s FAIL (2025-08-16 15:57:10 UTC) orbiter warning: context deadline exceeded Duration: 6.378436048s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 34.317309172s FAIL (2025-08-16 15:56:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.317330802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.248792303s FAIL (2025-08-16 15:56:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 77.90.31.13:4625 , piece_id=7DVOK2EHAELZCVFG3HUXHCHXPLI7ESHQ63XR5GG4ELA7JM5T4CWA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.363438365 (190h29m50.794518486s) [noise=true , address=79.127.201.215:4350 , piece_id=NVAHZQWQUCGCITF2NOTYAT6NIDRZ72MXF33G35WBQKBHKX6EWE3Q, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.363991755 (190h29m50.795071886s) [noise=true , address=104.36.82.38:4718 , piece_id=PSFTPQNXNV32NEWXVRRYKDQLL3KJGPZAMNKSYBATULW2NSBTNX6A, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.364295306 (190h29m50.795375437s) [noise=true , address=104.36.82.40:4965 , piece_id=UXH3MBTSPICTSSW45OBPZ2UXCRLRJJBZTDHPNXMGFQU6EN5BALBQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.364807425 (190h29m50.795887546s) [noise=true , address=104.36.82.44:4880 , piece_id=22A42P55EI7NC5PUNGXWUV62BGCQSGZXMODMQNIL5KAKZAMS3QEQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.364859745 (190h29m50.795939866s) [noise=true , address=77.90.31.10:4532 , piece_id=EMQSFDJOCRGRQIBPKRIGNBB5ATL47U7SRMQQJVU4REWFVBKWZ4RQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.365154129 (190h29m50.79623426s) [noise=true , address=77.90.31.21:4905 , piece_id=ID2G7SVDVQKGF6KMGRIVE6O4LLN45LC2G5L3K2LR6EZ5UFPO2SFQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.365432200 (190h29m50.796512332s) [noise=true , address=104.36.82.5:4872 , piece_id=5GYGH73AYZDYDIEF37Q6CMGNYSB6KQQV2SGKR5P5EZR7P77MFCTA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.365739909 (190h29m50.79682004s) [noise=true , address=77.90.31.11:4560 , piece_id=Y7Q6GN3YHCCF7L3PF3HAKHLOJDZLXYVIRKXB5H2IMO5FTTR4NBMA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.365855541 (190h29m50.796935672s) [noise=true , address=104.36.82.58:4269 , piece_id=DELQ27DKNBB7LQI2D7CPOLBW2HIOLUS6NJ2HIB7A6Z54O23WAPEA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.366093606 (190h29m50.797173737s) [noise=true , address=104.36.82.60:4349 , piece_id=2UUKZGAVTJ4QDC4MC5NVZSSTO7MD4273VO3TLM2LFT4UA7ZSSEMA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.366295192 (190h29m50.797375323s) [noise=true , address=66.23.193.175:4555 , piece_id=FZ2HT6MCZPRDUBYZ6ZWXQDB62TE3JIMHBNIOINYUH7PBCBPVYWPQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.366483281 (190h29m50.797563402s) [noise=true , address=45.91.116.9:4270 , piece_id=D3EA3MXYLASRK5REDEFO5OQOXS2XZYRSXTPUV7DXDJR2LOOODSVQ, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.366790810 (190h29m50.797870942s) [noise=true , address=104.36.82.4:4810 , piece_id=O7FEAEVC2OTE2N75COHKDXYM2O4C43HYFX5KXSTOSAGCGI6MTSIA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.366948061 (190h29m50.798028192s) [noise=true , address=77.90.31.19:4875 , piece_id=YLFV24HRJUARB2I334JE2THSEMH5JYUBTGQLY6ZPS7I22YRYXBRA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:56.367177150 (190h29m50.798257271s) [noise=true , address=45.91.116.12:4237 , piece_id=Y7BX6JZCU4D33Y3PL6DN7J5SVWIOU2SUTOF4YNKDEOKSYLVYWXNA, seg_pos={0 0}, upload=198178]: Uploading piece... Aug 16 15:55:57.504372545 (190h29m51.935452676s) [upload=198178]: Next returned nil segment Aug 16 15:55:57.504390810 (190h29m51.935470931s) [upload=198178]: Waiting for error group managing segments... Aug 16 15:56:01.008702725 (190h29m55.439782856s) [noise=true , address=79.127.201.215:4350 , piece_id=NVAHZQWQUCGCITF2NOTYAT6NIDRZ72MXF33G35WBQKBHKX6EWE3Q, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:05.890463091 (190h30m0.321543222s) [noise=true , address=66.23.193.175:4555 , piece_id=FZ2HT6MCZPRDUBYZ6ZWXQDB62TE3JIMHBNIOINYUH7PBCBPVYWPQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:09.898079384 (190h30m4.329159515s) [noise=true , address=77.90.31.19:4875 , piece_id=YLFV24HRJUARB2I334JE2THSEMH5JYUBTGQLY6ZPS7I22YRYXBRA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:11.865549285 (190h30m6.296629416s) [noise=true , address=77.90.31.13:4625 , piece_id=7DVOK2EHAELZCVFG3HUXHCHXPLI7ESHQ63XR5GG4ELA7JM5T4CWA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:15.500671803 (190h30m9.931751934s) [noise=true , address=77.90.31.21:4905 , piece_id=ID2G7SVDVQKGF6KMGRIVE6O4LLN45LC2G5L3K2LR6EZ5UFPO2SFQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:19.041009439 (190h30m13.47208956s) [noise=true , address=77.90.31.11:4560 , piece_id=Y7Q6GN3YHCCF7L3PF3HAKHLOJDZLXYVIRKXB5H2IMO5FTTR4NBMA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:23.040091375 (190h30m17.471171506s) [noise=true , address=45.91.116.9:4270 , piece_id=D3EA3MXYLASRK5REDEFO5OQOXS2XZYRSXTPUV7DXDJR2LOOODSVQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:24.852353577 (190h30m19.283433708s) [noise=true , address=45.91.116.12:4237 , piece_id=Y7BX6JZCU4D33Y3PL6DN7J5SVWIOU2SUTOF4YNKDEOKSYLVYWXNA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: <nil> Aug 16 15:56:55.228211284 (190h30m49.659291415s) [noise=true , address=104.36.82.38:4718 , piece_id=PSFTPQNXNV32NEWXVRRYKDQLL3KJGPZAMNKSYBATULW2NSBTNX6A, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98, address:104.36.82.38:4718): protocol: write tcp 188.166.225.170:58246->104.36.82.38:4718: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228304923 (190h30m49.659385054s) [noise=true , address=104.36.82.40:4965 , piece_id=UXH3MBTSPICTSSW45OBPZ2UXCRLRJJBZTDHPNXMGFQU6EN5BALBQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:1ToFTqjRAWvpRQzf6vyHKDpforPyb4LbDGcCVhRhkwiBGKihnD, address:104.36.82.40:4965): protocol: rpc: dial tcp 104.36.82.40:4965: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228334770 (190h30m49.659414901s) [noise=true , address=77.90.31.10:4532 , piece_id=EMQSFDJOCRGRQIBPKRIGNBB5ATL47U7SRMQQJVU4REWFVBKWZ4RQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:56:55.228372593 (190h30m49.659452724s) [noise=true , address=104.36.82.60:4349 , piece_id=2UUKZGAVTJ4QDC4MC5NVZSSTO7MD4273VO3TLM2LFT4UA7ZSSEMA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:104.36.82.60:4349): protocol: rpc: dial tcp 104.36.82.60:4349: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228440933 (190h30m49.659521065s) [noise=true , address=104.36.82.34:4549 , piece_id=YDFVMTJ4SULD7NYBBPDZBUEAO4A2E2UYSOA6L75Q4HFNQVLLM65Q, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:12tn1Sd8Gk3eHSmToBnPGjCyaAdQpUQrhiGe5cuecwBCpd7kEyT, address:104.36.82.34:4549): protocol: rpc: dial tcp 104.36.82.34:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228465280 (190h30m49.659545411s) [noise=true , address=104.36.82.5:4872 , piece_id=5GYGH73AYZDYDIEF37Q6CMGNYSB6KQQV2SGKR5P5EZR7P77MFCTA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: rpc: dial tcp 104.36.82.5:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228585200 (190h30m49.659665321s) [noise=true , address=104.36.82.58:4269 , piece_id=DELQ27DKNBB7LQI2D7CPOLBW2HIOLUS6NJ2HIB7A6Z54O23WAPEA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:12B63jT3ne2EnUiH7eRFWMpjzdirh6u7PwprmTUmDHPquoQwgk9, address:104.36.82.58:4269): protocol: rpc: dial tcp 104.36.82.58:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228623172 (190h30m49.659703304s) [noise=true , address=104.36.82.4:4810 , piece_id=O7FEAEVC2OTE2N75COHKDXYM2O4C43HYFX5KXSTOSAGCGI6MTSIA, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:104.36.82.4:4810): protocol: rpc: dial tcp 104.36.82.4:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228654332 (190h30m49.659734463s) [noise=true , address=104.36.82.44:4880 , piece_id=22A42P55EI7NC5PUNGXWUV62BGCQSGZXMODMQNIL5KAKZAMS3QEQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: write tcp 188.166.225.170:47156->104.36.82.44:4880: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:55.228740917 (190h30m49.659821049s) [noise=true , address=104.36.82.23:4446 , piece_id=OFUXWWIKPR52MOUOKVF5Y7QVGELMHQH5SICMS2ZSCHQ6TKDAHEEQ, seg_pos={0 0}, upload=198178]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:56:55.228770013 (190h30m49.659850144s) [seg_pos={0 0}, upload=198178]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:56:55.475189960 (190h30m49.906270081s) [upload=198178]: Done uploading segments... Aug 16 15:56:55.475216381 (190h30m49.906296512s) [upload=198178]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.248805579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 12.737936766s FAIL (2025-08-16 15:56:38 UTC) orbiter warning: context deadline exceeded Duration: 12.737957176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [...X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.249865937s FAIL (2025-08-16 15:56:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Aug 16 15:55:35.798936217 (190h29m30.230016348s) [upload=198175]: Starting upload Aug 16 15:55:35.798996133 (190h29m30.230076264s) [upload=198175]: Done starting upload Aug 16 15:55:35.799077018 (190h29m30.230157149s) [upload=198175]: Uploading segments... Aug 16 15:55:35.799260017 (190h29m30.230340138s) [upload=198175]: Got next segment. Inline: false Aug 16 15:55:35.799293611 (190h29m30.230373732s) [upload=198175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:55:35.799297660 (190h29m30.230377791s) [upload=198175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:55:36.259862057 (190h29m30.690942179s) [seg_pos={0 0}, upload=198175]: Begin upload segment... Aug 16 15:55:36.259907044 (190h29m30.690987175s) [seg_pos={0 0}, upload=198175]: Done begin upload segment. Aug 16 15:55:36.259939666 (190h29m30.691019797s) [noise=true , address=77.90.31.13:4635 , piece_id=KJPZXNGJVQ4LVAO32D2NFKEDQBMVTPZAGUDYLBXJJH3VH44BWTMA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.260039147 (190h29m30.691119278s) [noise=true , address=77.90.31.15:4706 , piece_id=Q6YUSPVDHGF5PB7OCMYOMMQE4TEN3IVPFHBWZGGOXS5IDH5O7FYA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.260433311 (190h29m30.691513442s) [noise=true , address=104.36.82.23:4448 , piece_id=R6D6GYZQ4YK4AVWLUJE7CCLBCGUJUACCU6QPVDM5OVANSWTYGC5A, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.260739738 (190h29m30.691819859s) [noise=true , address=77.90.31.21:4899 , piece_id=SMC2LI5PAAQCUZHXAQQ7DWALLY6OU757D3AAEXVUPXVD7E4L4SRA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.260875096 (190h29m30.691955227s) [noise=true , address=45.91.116.10:4110 , piece_id=R2D32JFVPCHJA3O2RF5AIIHC6UZDYAMWZSDQ5NYMWX6CP3IGIECA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261030934 (190h29m30.692111075s) [noise=true , address=79.127.220.101:4073 , piece_id=OUOGVSYMDC3M5HB3YYVTLFTZARJSCUPHBUW6SNXFODUH7WGP5K7Q, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261300300 (190h29m30.692380431s) [noise=true , address=64.49.13.139:4480 , piece_id=TF46QAEBUIITLJWLFKDO2KSSOR7ZRD7SEROQT66SGEGHHZMDWASQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261367167 (190h29m30.692447299s) [noise=true , address=104.36.82.16:4303 , piece_id=D66M76CJ772A66HXZBHV7NXFY3YKLQGDMYC5CIBEPFWFPJZEI7EA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261593862 (190h29m30.692673993s) [noise=true , address=104.36.82.5:4867 , piece_id=ML735XGQNLJCIQ2HUBDMENIFGRMDDTV5BUGKVRDHKLYIBHMRJEAQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261869339 (190h29m30.69294947s) [noise=true , address=104.36.82.17:4359 , piece_id=2XI54F37ZKBTWVAZMLB2T7QEY2YULZAGLANNB47GFLHRT4ZPOPIQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.261883435 (190h29m30.692963567s) [noise=true , address=66.23.193.175:4556 , piece_id=PDJBSFKNCBUTG5OOVYK4S47EQJOV2U4XEV2SLMOMSPBYT7D54UYA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.262159794 (190h29m30.693239925s) [noise=true , address=104.36.82.44:4897 , piece_id=NVR2IFSJWURYPYUM6VBGL5URTR3WLKITZ2H2ENJQKHWTK4O6BQMQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.262348415 (190h29m30.693428546s) [noise=true , address=77.90.31.18:4840 , piece_id=73YKVPVROMD7KJEPQSCUFMEJQYFK6UQBDZ23JG4SFJUQVCJ4WFOQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.262382250 (190h29m30.693462381s) [noise=true , address=104.36.82.38:4704 , piece_id=3M5M3RBLID7RN6A5DPU4NHGMVJ7QJJQUKMFZ6GPNBZJMJU23S33A, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.262652517 (190h29m30.693732638s) [noise=true , address=104.36.82.60:4346 , piece_id=HF44PZATZQW6R6M35KSYZEZT6A3YD545MXMDUNZPUUUJVIGGDZVA, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.262802364 (190h29m30.693882495s) [noise=true , address=104.36.82.35:4611 , piece_id=JKMB77RRFKZHKXVB3D2FSYIX3LYO2CYGUWYLQA67BNW2Z2VKCM6A, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.264416592 (190h29m30.695496723s) [noise=true , address=77.90.31.12:4597 , piece_id=RHUNTLUUYCR2O6XH274VWNPZC225A6LHL6GNFUFEU2DLY6LLRTVQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:36.264789476 (190h29m30.695869607s) [noise=true , address=104.36.82.39:4742 , piece_id=X5CG3THSFJCLN2EGEGZ725BJJ376IQYB5FUASFF6PO6AARI32TQQ, seg_pos={0 0}, upload=198175]: Uploading piece... Aug 16 15:55:37.076401013 (190h29m31.507481144s) [noise=true , address=66.23.193.175:4556 , piece_id=PDJBSFKNCBUTG5OOVYK4S47EQJOV2U4XEV2SLMOMSPBYT7D54UYA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12RHXPzzH8G9xGyABLH7hNnmkZuDdWpoiQkoTxNBzdzNvNQhN29, address:66.23.193.175:4556): protocol: storage node overloaded, request limit: 2000 Aug 16 15:55:37.417199804 (190h29m31.848279935s) [upload=198175]: Next returned nil segment Aug 16 15:55:37.417218799 (190h29m31.84829893s) [upload=198175]: Waiting for error group managing segments... Aug 16 15:55:37.762729166 (190h29m32.193809297s) [noise=true , address=64.49.13.139:4480 , piece_id=TF46QAEBUIITLJWLFKDO2KSSOR7ZRD7SEROQT66SGEGHHZMDWASQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:38.107595558 (190h29m32.538675679s) [noise=true , address=79.127.220.101:4073 , piece_id=OUOGVSYMDC3M5HB3YYVTLFTZARJSCUPHBUW6SNXFODUH7WGP5K7Q, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:41.220343544 (190h29m35.651423675s) [noise=true , address=77.90.31.12:4597 , piece_id=RHUNTLUUYCR2O6XH274VWNPZC225A6LHL6GNFUFEU2DLY6LLRTVQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:41.824850581 (190h29m36.255930702s) [noise=true , address=77.90.31.21:4899 , piece_id=SMC2LI5PAAQCUZHXAQQ7DWALLY6OU757D3AAEXVUPXVD7E4L4SRA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:48.444702214 (190h29m42.875782355s) [noise=true , address=77.90.31.13:4635 , piece_id=KJPZXNGJVQ4LVAO32D2NFKEDQBMVTPZAGUDYLBXJJH3VH44BWTMA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:48.665981617 (190h29m43.097061748s) [noise=true , address=77.90.31.15:4706 , piece_id=Q6YUSPVDHGF5PB7OCMYOMMQE4TEN3IVPFHBWZGGOXS5IDH5O7FYA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:52.683448829 (190h29m47.11452896s) [noise=true , address=77.90.31.18:4840 , piece_id=73YKVPVROMD7KJEPQSCUFMEJQYFK6UQBDZ23JG4SFJUQVCJ4WFOQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:55:59.969091576 (190h29m54.400171707s) [noise=true , address=45.91.116.10:4110 , piece_id=R2D32JFVPCHJA3O2RF5AIIHC6UZDYAMWZSDQ5NYMWX6CP3IGIECA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: <nil> Aug 16 15:56:35.136824669 (190h30m29.5679048s) [noise=true , address=104.36.82.5:4867 , piece_id=ML735XGQNLJCIQ2HUBDMENIFGRMDDTV5BUGKVRDHKLYIBHMRJEAQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:104.36.82.5:4867): protocol: rpc: dial tcp 104.36.82.5:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.136872881 (190h30m29.567953002s) [noise=true , address=104.36.82.17:4359 , piece_id=2XI54F37ZKBTWVAZMLB2T7QEY2YULZAGLANNB47GFLHRT4ZPOPIQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:1MdZDBkhh7vwffrHksdiixu99PiTgmmNAzdUo39YGaEtpuAHdR, address:104.36.82.17:4359): protocol: rpc: dial tcp 104.36.82.17:4359: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.136911495 (190h30m29.567991626s) [noise=true , address=104.36.82.38:4704 , piece_id=3M5M3RBLID7RN6A5DPU4NHGMVJ7QJJQUKMFZ6GPNBZJMJU23S33A, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:1225CoSQvGuiWjJZWaXhvvq4gxzMymZqUb9fBrfLWw4RekwAEhp, address:104.36.82.38:4704): protocol: rpc: dial tcp 104.36.82.38:4704: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.136958896 (190h30m29.568039027s) [noise=true , address=104.36.82.35:4611 , piece_id=JKMB77RRFKZHKXVB3D2FSYIX3LYO2CYGUWYLQA67BNW2Z2VKCM6A, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc, address:104.36.82.35:4611): protocol: rpc: dial tcp 104.36.82.35:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.136974937 (190h30m29.568055068s) [noise=true , address=104.36.82.39:4742 , piece_id=X5CG3THSFJCLN2EGEGZ725BJJ376IQYB5FUASFF6PO6AARI32TQQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12V9GZR1rr6VfEsK83Bonq2HyceXLcAKnYfoQmA1Qvt5h9u8sik, address:104.36.82.39:4742): protocol: rpc: dial tcp 104.36.82.39:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.136989144 (190h30m29.568069275s) [noise=true , address=104.36.82.16:4303 , piece_id=D66M76CJ772A66HXZBHV7NXFY3YKLQGDMYC5CIBEPFWFPJZEI7EA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: rpc: dial tcp 104.36.82.16:4303: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.137026846 (190h30m29.568106967s) [noise=true , address=104.36.82.60:4346 , piece_id=HF44PZATZQW6R6M35KSYZEZT6A3YD545MXMDUNZPUUUJVIGGDZVA, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72, address:104.36.82.60:4346): protocol: rpc: dial tcp 104.36.82.60:4346: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.137035342 (190h30m29.568115473s) [noise=true , address=104.36.82.23:4448 , piece_id=R6D6GYZQ4YK4AVWLUJE7CCLBCGUJUACCU6QPVDM5OVANSWTYGC5A, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i, address:104.36.82.23:4448): protocol: write tcp 188.166.225.170:59200->104.36.82.23:4448: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.137109945 (190h30m29.568190267s) [noise=true , address=104.36.82.44:4897 , piece_id=NVR2IFSJWURYPYUM6VBGL5URTR3WLKITZ2H2ENJQKHWTK4O6BQMQ, seg_pos={0 0}, upload=198175]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: write tcp 188.166.225.170:59528->104.36.82.44:4897: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:56:35.137143950 (190h30m29.568224081s) [seg_pos={0 0}, upload=198175]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:56:35.384942154 (190h30m29.816022285s) [upload=198175]: Done uploading segments... Aug 16 15:56:35.384960159 (190h30m29.81604029s) [upload=198175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.24991337s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..................X......X..XXXXXXXXXX.XX.XXXXXX......X.....] duration: 7.115841344s FAIL (2025-08-16 15:56:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.115866103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [............X.X..X.....X......................X.............] duration: 1.931922065s FAIL (2025-08-16 15:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.931942062s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 25.199005885s FAIL (2025-08-16 15:56:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.199029852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...................X......X..XXXXXXXXXX.XX.XXXXXX......X....] duration: 27.978326996s FAIL (2025-08-16 15:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 27.978357104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230137412s FAIL (2025-08-16 15:55:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ing piece... Aug 16 15:54:55.858970333 (190h28m50.290050464s) [noise=true , address=45.91.116.4:4950 , piece_id=WWZORQJNQGVFSZ5E4SQKS3PWNETEV46M2OZBHDGXY5WRRQEY2BMQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.859316857 (190h28m50.290396988s) [noise=true , address=77.90.31.18:4809 , piece_id=TWCT4JA73QLUZ2JUZYBUUHGMB2RSPYUCIJHJCVC4O6HWLHQDMBEA, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.859424032 (190h28m50.290504163s) [noise=true , address=79.127.220.101:4060 , piece_id=AYNUJVGOGSKTQP6KZGFZWUUPL6RB54K2QM5A45NVSOXTZ46ZEDAQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.859573808 (190h28m50.290653939s) [noise=true , address=104.36.82.44:4876 , piece_id=P7SMRDXEX4TZ5DPHGAF7FBS3AJJVJCZRKE4JSPDCMLDZEQAW6NAQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.859832113 (190h28m50.290912244s) [noise=true , address=104.36.82.16:4328 , piece_id=C74PQUZJDYBILVD3VBPVHAVYOQEJKBY2UMJTJD2FUSMW7CQTREYQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.860143979 (190h28m50.29122411s) [noise=true , address=77.90.31.19:4868 , piece_id=YUU7I6GU7YNQZNMD3PKYV4GN4RCIMYGVS226JBNONRDCFJAB2O6Q, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.860423794 (190h28m50.291503925s) [noise=true , address=77.90.31.11:4576 , piece_id=KBVGOJEGCMYQSH2BB5567HQCER7PKSPDL4KMJ5FYP6DP4JTYZEHA, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.860638185 (190h28m50.291718316s) [noise=true , address=66.23.193.175:4560 , piece_id=5H7HW7KFQOXVSVSOUOL2QIW3QXEUDANTMZNAWEXCK5K2BE26KLKQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.860860610 (190h28m50.291940741s) [noise=true , address=77.90.31.21:4912 , piece_id=44JZIEDIBRSGQWVNAEQHKQE4MEVTMLPQFWP3YI45I4MOTNDAD66A, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.861780000 (190h28m50.292860131s) [noise=true , address=79.127.201.215:4357 , piece_id=K3MHHUUMHYAVJKHDBEUMDNJPCZT4XDKJ4WHNWLHZ5C63AKPFCOAQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.862014939 (190h28m50.29309507s) [noise=true , address=104.36.82.60:4359 , piece_id=XKS6BI6X7IYZVODTRZDRX7RFQFE263VLTSINGWXQVNOCFP6BQXQA, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.862229970 (190h28m50.293310101s) [noise=true , address=104.36.82.34:4578 , piece_id=UT6LD4XPIZ5FDPLYH5MDF4DDNTFGY4NKQCOO2B6ZVK324PPXXLIQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.862439652 (190h28m50.293519783s) [noise=true , address=104.36.82.12:4162 , piece_id=PUXHJ76TRP4PQB6BXEAUKMA54H52RYB6IMLX6P6JSKTXTW72YKWA, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.862867199 (190h28m50.29394734s) [noise=true , address=104.36.82.39:4725 , piece_id=LA3FFUYUR3K24ZUMJU2ME35VJNBD3C655BXLWFYV27BXPJZ6GBIA, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.863148699 (190h28m50.29422883s) [noise=true , address=79.127.220.101:4071 , piece_id=JTQ4GX5WPHDG3ITNBYGFUWV7SOY5VJKFMVQA2C7DW62GJ77JXCEQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.863355925 (190h28m50.294436056s) [noise=true , address=104.36.82.5:4860 , piece_id=6HF26C7YP6HM7MARBGN6NZHEBEM3VLHDKQXW3ICQVVLHDWZNCQHQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:55.863611804 (190h28m50.294691935s) [noise=true , address=104.36.82.59:4326 , piece_id=KYXLCLRV3XAYLQFZ5F2L74ZT5BHDH6NJCLXXME7TG2ISLYGDBCBQ, seg_pos={0 0}, upload=198162]: Uploading piece... Aug 16 15:54:56.958783984 (190h28m51.389864105s) [upload=198162]: Next returned nil segment Aug 16 15:54:56.958801247 (190h28m51.389881378s) [upload=198162]: Waiting for error group managing segments... Aug 16 15:54:58.509647878 (190h28m52.940728009s) [noise=true , address=77.90.31.19:4868 , piece_id=YUU7I6GU7YNQZNMD3PKYV4GN4RCIMYGVS226JBNONRDCFJAB2O6Q, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:54:58.620827196 (190h28m53.051907327s) [noise=true , address=45.91.116.4:4950 , piece_id=WWZORQJNQGVFSZ5E4SQKS3PWNETEV46M2OZBHDGXY5WRRQEY2BMQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:54:58.772067178 (190h28m53.203147309s) [noise=true , address=79.127.220.101:4071 , piece_id=JTQ4GX5WPHDG3ITNBYGFUWV7SOY5VJKFMVQA2C7DW62GJ77JXCEQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:54:58.878345019 (190h28m53.30942515s) [noise=true , address=77.90.31.21:4912 , piece_id=44JZIEDIBRSGQWVNAEQHKQE4MEVTMLPQFWP3YI45I4MOTNDAD66A, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:54:59.754029543 (190h28m54.185109684s) [noise=true , address=77.90.31.11:4576 , piece_id=KBVGOJEGCMYQSH2BB5567HQCER7PKSPDL4KMJ5FYP6DP4JTYZEHA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:54:59.856538854 (190h28m54.287618975s) [noise=true , address=79.127.220.101:4060 , piece_id=AYNUJVGOGSKTQP6KZGFZWUUPL6RB54K2QM5A45NVSOXTZ46ZEDAQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:55:02.257182185 (190h28m56.688262326s) [noise=true , address=79.127.201.215:4357 , piece_id=K3MHHUUMHYAVJKHDBEUMDNJPCZT4XDKJ4WHNWLHZ5C63AKPFCOAQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:55:16.117182318 (190h29m10.548262449s) [noise=true , address=104.36.82.12:4162 , piece_id=PUXHJ76TRP4PQB6BXEAUKMA54H52RYB6IMLX6P6JSKTXTW72YKWA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:55:25.490206571 (190h29m19.921286702s) [noise=true , address=66.23.193.175:4560 , piece_id=5H7HW7KFQOXVSVSOUOL2QIW3QXEUDANTMZNAWEXCK5K2BE26KLKQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:55:38.987101165 (190h29m33.418181296s) [noise=true , address=77.90.31.18:4809 , piece_id=TWCT4JA73QLUZ2JUZYBUUHGMB2RSPYUCIJHJCVC4O6HWLHQDMBEA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: <nil> Aug 16 15:55:54.763610204 (190h29m49.194690335s) [noise=true , address=104.36.82.16:4328 , piece_id=C74PQUZJDYBILVD3VBPVHAVYOQEJKBY2UMJTJD2FUSMW7CQTREYQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:4328): protocol: write tcp 188.166.225.170:59564->104.36.82.16:4328: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763675660 (190h29m49.194755791s) [noise=true , address=104.36.82.60:4359 , piece_id=XKS6BI6X7IYZVODTRZDRX7RFQFE263VLTSINGWXQVNOCFP6BQXQA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV, address:104.36.82.60:4359): protocol: rpc: dial tcp 104.36.82.60:4359: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763712209 (190h29m49.19479234s) [noise=true , address=104.36.82.39:4725 , piece_id=LA3FFUYUR3K24ZUMJU2ME35VJNBD3C655BXLWFYV27BXPJZ6GBIA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:104.36.82.39:4725): protocol: write tcp 188.166.225.170:53624->104.36.82.39:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763770791 (190h29m49.194850922s) [noise=true , address=104.36.82.35:4581 , piece_id=TUNJQWAQZMMXZFIM65QEWZAF6EKDV2L7M4KCECITNOFUSMPXF2LA, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:104.36.82.35:4581): protocol: rpc: dial tcp 104.36.82.35:4581: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763813774 (190h29m49.194893905s) [noise=true , address=104.36.82.44:4876 , piece_id=P7SMRDXEX4TZ5DPHGAF7FBS3AJJVJCZRKE4JSPDCMLDZEQAW6NAQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC, address:104.36.82.44:4876): protocol: rpc: dial tcp 104.36.82.44:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763844963 (190h29m49.194925095s) [noise=true , address=104.36.82.34:4578 , piece_id=UT6LD4XPIZ5FDPLYH5MDF4DDNTFGY4NKQCOO2B6ZVK324PPXXLIQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:1poqPvwwjWwAAhkJ5WCDavg9oKxbJrf9xktnYA9DB6o9BV1G49, address:104.36.82.34:4578): protocol: rpc: dial tcp 104.36.82.34:4578: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763897905 (190h29m49.194978036s) [noise=true , address=104.36.82.59:4326 , piece_id=KYXLCLRV3XAYLQFZ5F2L74ZT5BHDH6NJCLXXME7TG2ISLYGDBCBQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:12Ys2boqq8tJDTRRSGV47s1uhZqA68LzZuQ3HwpKwghUPc5Dwy7, address:104.36.82.59:4326): protocol: rpc: dial tcp 104.36.82.59:4326: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763972908 (190h29m49.195053039s) [noise=true , address=104.36.82.5:4860 , piece_id=6HF26C7YP6HM7MARBGN6NZHEBEM3VLHDKQXW3ICQVVLHDWZNCQHQ, seg_pos={0 0}, upload=198162]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: write tcp 188.166.225.170:38368->104.36.82.5:4860: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:54.763988869 (190h29m49.195069s) [seg_pos={0 0}, upload=198162]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:55:54.992023698 (190h29m49.423103819s) [upload=198162]: Done uploading segments... Aug 16 15:55:54.992047875 (190h29m49.423128006s) [upload=198162]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230148053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [....X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237161267s FAIL (2025-08-16 15:55:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:54:35.323577058 (190h28m29.754657189s) [upload=198154]: Starting upload Aug 16 15:54:35.323703159 (190h28m29.75478329s) [upload=198154]: Done starting upload Aug 16 15:54:35.323860239 (190h28m29.75494038s) [upload=198154]: Uploading segments... Aug 16 15:54:35.324090050 (190h28m29.755170171s) [upload=198154]: Got next segment. Inline: false Aug 16 15:54:35.324124345 (190h28m29.755204476s) [upload=198154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:54:35.324128813 (190h28m29.755208944s) [upload=198154]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:54:35.779951961 (190h28m30.211032092s) [seg_pos={0 0}, upload=198154]: Begin upload segment... Aug 16 15:54:35.779996225 (190h28m30.211076346s) [seg_pos={0 0}, upload=198154]: Done begin upload segment. Aug 16 15:54:35.780020742 (190h28m30.211100863s) [noise=true , address=45.91.116.11:4123 , piece_id=Q3RMLPEBSUNCTNPD75M3T7PKOASNF2PAV64H37EEZ7EEINHF3VWQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.780051180 (190h28m30.211131311s) [noise=true , address=104.36.82.58:4298 , piece_id=JEUDRUOOKWWZPIBN3GV4OFFSOA7LNTAOVP563PD7K62AILILSR7A, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.780501963 (190h28m30.211582094s) [noise=true , address=77.90.31.21:4887 , piece_id=5TKYDTITGJRICWUFHYE53PCEEN7DZO22ONF3AATFZHPIRFOII6SA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.780573389 (190h28m30.21165355s) [noise=true , address=104.36.82.60:4361 , piece_id=5E6K7MQKVM553IDS3RMT63YMUYWO7QYUHZ2LSH2AV5O5YZO5X5UQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.780790715 (190h28m30.211870846s) [noise=true , address=79.127.201.215:4345 , piece_id=KUYTIU5JFMZKJQQ5QCEWXRJ44E23KHCOQJTJFO63AHU2WDSACQNA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.780909022 (190h28m30.211989153s) [noise=true , address=77.90.31.15:4702 , piece_id=AN2A7CE7IZ2QXQT6CFMFBECDEZ426CLYIC2R7LCSEOE66J66SXUA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781087093 (190h28m30.212167224s) [noise=true , address=77.90.31.18:4813 , piece_id=H7PWPK2J3Q3JRA2XQ3CS7GE5V3BQ7A7RZWYT5AUP4CTUBFTQFLJQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781187615 (190h28m30.212267746s) [noise=true , address=45.91.116.10:4110 , piece_id=DFP6UM4G6MMHZ4NLW53HGZYX7JNYQBGPNN5PBQMKCQVT7ZHVACHA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781420871 (190h28m30.212501002s) [noise=true , address=77.90.31.11:4563 , piece_id=TFJSVSZYEIAEUJFU7FQF3YHU2N2XIEPDT3ALPZXBTCOSBQEVBCPA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781477349 (190h28m30.21255747s) [noise=true , address=104.36.82.35:4607 , piece_id=C6K3PKMFGGNMOGH53RUH5L3IR3JF5WY3F6ZTC2L7VNJKURBASVYQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781776682 (190h28m30.212856813s) [noise=true , address=104.36.82.44:4892 , piece_id=WZZ2E4MSON7E76V6JXKE7GLFRPLQJMAN7DW6URMEOYBPWQSEETCQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.781781781 (190h28m30.212861912s) [noise=true , address=77.90.31.19:4851 , piece_id=V4PEX6PXJSVSJ5POZKP7SEWIKXCNANO5MK5LH27GHOSQY3LFXLPA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.782068300 (190h28m30.213148431s) [noise=true , address=104.36.82.3:4782 , piece_id=H47JYY7JBOKF7FKRMWOYFFMCL7V5QGSSB66PG5HRCZS4JMDNUHXQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.782145638 (190h28m30.213225769s) [noise=true , address=104.36.82.23:4452 , piece_id=EEUHAU62KEHA6KATLDTGLSSF36KTCYBIDRMSVNEXLDZECFDICJ6Q, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.782371260 (190h28m30.213451381s) [noise=true , address=104.36.82.5:4855 , piece_id=RPMZCZSJQQN452W2LQZ67IKTC6NY3NNVUXC3TCD2BPQ4XSRVEKRQ, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.782581983 (190h28m30.213662114s) [noise=true , address=77.90.31.10:4524 , piece_id=MN2V3ZTL6CUHONBXVVWNTEMPRZTPQZX6OJ4HAEBBGQC5YEW6RS3Q, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.782800301 (190h28m30.213880432s) [noise=true , address=66.23.193.175:4564 , piece_id=LZ6G2OYLQSROTEZRJTBUXJQAEJ3COFFGNT6TJDFORGQDWVFRWE4Q, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:35.785988291 (190h28m30.217068412s) [noise=true , address=104.36.82.18:4375 , piece_id=6ZJRABNGEEA6JUUHAHGQL5W6ML7DFCEWERSNCNQ62IR7X2JTJEOA, seg_pos={0 0}, upload=198154]: Uploading piece... Aug 16 15:54:37.146644162 (190h28m31.577724293s) [upload=198154]: Next returned nil segment Aug 16 15:54:37.146666766 (190h28m31.577746897s) [upload=198154]: Waiting for error group managing segments... Aug 16 15:54:38.029322688 (190h28m32.460402819s) [noise=true , address=77.90.31.19:4851 , piece_id=V4PEX6PXJSVSJ5POZKP7SEWIKXCNANO5MK5LH27GHOSQY3LFXLPA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:38.327193037 (190h28m32.758273168s) [noise=true , address=77.90.31.15:4702 , piece_id=AN2A7CE7IZ2QXQT6CFMFBECDEZ426CLYIC2R7LCSEOE66J66SXUA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:38.354891375 (190h28m32.785971496s) [noise=true , address=77.90.31.11:4563 , piece_id=TFJSVSZYEIAEUJFU7FQF3YHU2N2XIEPDT3ALPZXBTCOSBQEVBCPA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:39.519944734 (190h28m33.951024855s) [noise=true , address=45.91.116.11:4123 , piece_id=Q3RMLPEBSUNCTNPD75M3T7PKOASNF2PAV64H37EEZ7EEINHF3VWQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:44.024345247 (190h28m38.455425378s) [noise=true , address=79.127.201.215:4345 , piece_id=KUYTIU5JFMZKJQQ5QCEWXRJ44E23KHCOQJTJFO63AHU2WDSACQNA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:44.718838240 (190h28m39.149918371s) [noise=true , address=45.91.116.10:4110 , piece_id=DFP6UM4G6MMHZ4NLW53HGZYX7JNYQBGPNN5PBQMKCQVT7ZHVACHA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:47.488686814 (190h28m41.919766945s) [noise=true , address=77.90.31.10:4524 , piece_id=MN2V3ZTL6CUHONBXVVWNTEMPRZTPQZX6OJ4HAEBBGQC5YEW6RS3Q, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:54:54.843817101 (190h28m49.274897232s) [noise=true , address=77.90.31.21:4887 , piece_id=5TKYDTITGJRICWUFHYE53PCEEN7DZO22ONF3AATFZHPIRFOII6SA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:55:03.269238592 (190h28m57.700318724s) [noise=true , address=66.23.193.175:4564 , piece_id=LZ6G2OYLQSROTEZRJTBUXJQAEJ3COFFGNT6TJDFORGQDWVFRWE4Q, seg_pos={0 0}, upload=198154]: Done uploading piece. err: <nil> Aug 16 15:55:34.660380500 (190h29m29.091460621s) [noise=true , address=104.36.82.44:4892 , piece_id=WZZ2E4MSON7E76V6JXKE7GLFRPLQJMAN7DW6URMEOYBPWQSEETCQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: rpc: dial tcp 104.36.82.44:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660428422 (190h29m29.091508553s) [noise=true , address=104.36.82.18:4375 , piece_id=6ZJRABNGEEA6JUUHAHGQL5W6ML7DFCEWERSNCNQ62IR7X2JTJEOA, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:12pHU3Xp1sDgnmYMcbWZD2DivxdEUzqnBVJaiRaQipuRy2d75u5, address:104.36.82.18:4375): protocol: rpc: dial tcp 104.36.82.18:4375: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660514536 (190h29m29.091594657s) [noise=true , address=104.36.82.58:4298 , piece_id=JEUDRUOOKWWZPIBN3GV4OFFSOA7LNTAOVP563PD7K62AILILSR7A, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:1MWsyGhKkDhjngUDDRvxmhQddbNnRZ1A3mpzEtjLs9ZcRC5LBz, address:104.36.82.58:4298): protocol: rpc: dial tcp 104.36.82.58:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660574120 (190h29m29.091654251s) [noise=true , address=77.90.31.18:4813 , piece_id=H7PWPK2J3Q3JRA2XQ3CS7GE5V3BQ7A7RZWYT5AUP4CTUBFTQFLJQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: write tcp 188.166.225.170:50570->77.90.31.18:4813: use of closed network connection; write tcp 188.166.225.170:50570->77.90.31.18:4813: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50570->77.90.31.18:4813: use of closed network connection Aug 16 15:55:34.660607113 (190h29m29.091687244s) [noise=true , address=104.36.82.60:4361 , piece_id=5E6K7MQKVM553IDS3RMT63YMUYWO7QYUHZ2LSH2AV5O5YZO5X5UQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: rpc: dial tcp 104.36.82.60:4361: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660646589 (190h29m29.09172672s) [noise=true , address=104.36.82.35:4607 , piece_id=C6K3PKMFGGNMOGH53RUH5L3IR3JF5WY3F6ZTC2L7VNJKURBASVYQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660654354 (190h29m29.091734485s) [noise=true , address=104.36.82.3:4782 , piece_id=H47JYY7JBOKF7FKRMWOYFFMCL7V5QGSSB66PG5HRCZS4JMDNUHXQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:104.36.82.3:4782): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:55:34.660692416 (190h29m29.091772547s) [noise=true , address=104.36.82.23:4452 , piece_id=EEUHAU62KEHA6KATLDTGLSSF36KTCYBIDRMSVNEXLDZECFDICJ6Q, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: rpc: dial tcp 104.36.82.23:4452: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660698468 (190h29m29.091778599s) [noise=true , address=104.36.82.5:4855 , piece_id=RPMZCZSJQQN452W2LQZ67IKTC6NY3NNVUXC3TCD2BPQ4XSRVEKRQ, seg_pos={0 0}, upload=198154]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:40332->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:55:34.660715020 (190h29m29.091795151s) [seg_pos={0 0}, upload=198154]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:55:34.895335018 (190h29m29.326415149s) [upload=198154]: Done uploading segments... Aug 16 15:55:34.895358062 (190h29m29.326438193s) [upload=198154]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.237176124s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.030417876s FAIL (2025-08-16 15:55:25 UTC) orbiter warning: context deadline exceeded Duration: 7.03044071s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 27.510452875s FAIL (2025-08-16 15:55:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.510464428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.............X.X..X.....X......................X............] duration: 2.489422221s FAIL (2025-08-16 15:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.489448411s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 45.413318886s FAIL (2025-08-16 15:54:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.413341398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236225136s FAIL (2025-08-16 15:54:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== {0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.151706200 (190h27m49.582786331s) [noise=true , address=77.90.31.15:4711 , piece_id=HX7L2HOXVD3LRAN2W5XWNPQ3DWJWGS34AIAU7WP2M7DYUCZJLVYQ, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.151811882 (190h27m49.582892003s) [noise=true , address=104.36.82.36:4623 , piece_id=SD7MY6YZ6XO7QYL2H2GUXCHKOTP7Z73E2GROTBXEQDA3YJZVDZIA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152008198 (190h27m49.583088329s) [noise=true , address=45.91.116.14:4210 , piece_id=DNZKXPLB4CW4DQTATBKFVQ4ECDJCJGE2EI7GVMUVV7OK44EP275A, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152063033 (190h27m49.583143164s) [noise=true , address=45.91.116.5:4315 , piece_id=ERKC7XTJFXJO5QGG4BWZXMSRHUYVHOL6BSISGGKFEQX4PXAFFVXA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152356283 (190h27m49.583436414s) [noise=true , address=104.36.82.44:4898 , piece_id=YZHSVR4UJCYSJXXFIVRSETD7IGNYZPWNRQLBZ3DLGO532SNFEFIQ, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152368617 (190h27m49.583448738s) [noise=true , address=104.36.82.59:4318 , piece_id=NOVBFHNY2JIBGX6VGJOOEWCENYTMXV5E4GSHPSEJLJOGGVIDMYXA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152635638 (190h27m49.583715769s) [noise=true , address=104.36.82.23:4451 , piece_id=IDKS5VWOOD572L6BWNSGELOVUWXBAY2VS35CJDRZFEEVKRSQI2HA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152720691 (190h27m49.583800822s) [noise=true , address=104.36.82.3:4788 , piece_id=YUHE4S5XCSCSYFQ2D5ACYHVJRIPXVAV4RO6PHZOWRZWC6N7J2LLA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.152947825 (190h27m49.584027946s) [noise=true , address=77.90.31.10:4539 , piece_id=Y4HIF54QZVYKCF3NX4VSSNSXJ35S34YJDLSUIEM7TOHZEUXFCNUA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.153146515 (190h27m49.584226646s) [noise=true , address=77.90.31.13:4658 , piece_id=J437NWYIB55AT73P4CZWRCI2PTVHU2UTLOPVUFXCXDEMA3TH4NZA, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.153340206 (190h27m49.584420337s) [noise=true , address=66.23.193.175:4562 , piece_id=2BTA3743H2I6SU3FOMUHSQXP5UWSP2KKMGLXVWNDBK5VLPWNNE3Q, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.153543616 (190h27m49.584623737s) [noise=true , address=77.90.31.14:4667 , piece_id=EJ7X3Y7IOMM34N7NCJ7XLMGJ5VP7RHTZ4LZJ2QDQ4UG7OC22FV6A, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.153741514 (190h27m49.584821665s) [noise=true , address=104.36.82.35:4602 , piece_id=RJ7ELTYV7AEHVCO7YHNFLNPYWVVEXPDSBB3TTXF4RITPKJQQ54KQ, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.154029034 (190h27m49.585109155s) [noise=true , address=104.36.82.5:4855 , piece_id=XNLFJJ52NKBK2T3L7CP6IKUPUA4NY4YRWQIGPOLZ4L5ZQ6PHLVCQ, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:55.154221502 (190h27m49.585301633s) [noise=true , address=79.127.201.215:4353 , piece_id=RZYV7YKJUG5FB7WM2O3UF7UOXRN33M6L7BKJD3OUUCEEMMJI5KGQ, seg_pos={0 0}, upload=198147]: Uploading piece... Aug 16 15:53:56.402018757 (190h27m50.833098888s) [noise=true , address=156.146.43.225:4457 , piece_id=CY4U2FRXFGHUHMQ4PP3RXBCD55AYSM4YCSQPVY4L4EYZNOV5NPMA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:12v4SYj7tY4VFY9LqBpQDrpSmAbFEtCpPTcasMwtcSUtiQWMnJT, address:156.146.43.225:4457): protocol: storage node overloaded, request limit: 2000 Aug 16 15:53:56.690404421 (190h27m51.121484542s) [upload=198147]: Next returned nil segment Aug 16 15:53:56.690421413 (190h27m51.121501545s) [upload=198147]: Waiting for error group managing segments... Aug 16 15:53:57.133239748 (190h27m51.564319879s) [noise=true , address=77.90.31.15:4711 , piece_id=HX7L2HOXVD3LRAN2W5XWNPQ3DWJWGS34AIAU7WP2M7DYUCZJLVYQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:53:57.366541776 (190h27m51.797621907s) [noise=true , address=77.90.31.13:4658 , piece_id=J437NWYIB55AT73P4CZWRCI2PTVHU2UTLOPVUFXCXDEMA3TH4NZA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:53:57.398237022 (190h27m51.829317153s) [noise=true , address=45.91.116.5:4315 , piece_id=ERKC7XTJFXJO5QGG4BWZXMSRHUYVHOL6BSISGGKFEQX4PXAFFVXA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:53:57.405009670 (190h27m51.836089801s) [noise=true , address=45.91.116.14:4210 , piece_id=DNZKXPLB4CW4DQTATBKFVQ4ECDJCJGE2EI7GVMUVV7OK44EP275A, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:53:57.877855437 (190h27m52.308935568s) [noise=true , address=77.90.31.14:4667 , piece_id=EJ7X3Y7IOMM34N7NCJ7XLMGJ5VP7RHTZ4LZJ2QDQ4UG7OC22FV6A, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:53:58.863351275 (190h27m53.294431406s) [noise=true , address=79.127.201.215:4353 , piece_id=RZYV7YKJUG5FB7WM2O3UF7UOXRN33M6L7BKJD3OUUCEEMMJI5KGQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:54:09.133661607 (190h28m3.564741728s) [noise=true , address=66.23.193.175:4562 , piece_id=2BTA3743H2I6SU3FOMUHSQXP5UWSP2KKMGLXVWNDBK5VLPWNNE3Q, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:54:09.709229922 (190h28m4.140310053s) [noise=true , address=77.90.31.10:4539 , piece_id=Y4HIF54QZVYKCF3NX4VSSNSXJ35S34YJDLSUIEM7TOHZEUXFCNUA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: <nil> Aug 16 15:54:54.048622113 (190h28m48.479702244s) [noise=true , address=104.36.82.5:4855 , piece_id=XNLFJJ52NKBK2T3L7CP6IKUPUA4NY4YRWQIGPOLZ4L5ZQ6PHLVCQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: rpc: dial tcp 104.36.82.5:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048671578 (190h28m48.479751709s) [noise=true , address=104.36.82.4:4817 , piece_id=S3I5A64ZKZWKPLSCELUPITO4BAVIHP4ZN35OUDUFHEAFXML47GNQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz, address:104.36.82.4:4817): protocol: rpc: dial tcp 104.36.82.4:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048708099 (190h28m48.47978823s) [noise=true , address=104.36.82.23:4451 , piece_id=IDKS5VWOOD572L6BWNSGELOVUWXBAY2VS35CJDRZFEEVKRSQI2HA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: rpc: dial tcp 104.36.82.23:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048808700 (190h28m48.479888831s) [noise=true , address=104.36.82.44:4898 , piece_id=YZHSVR4UJCYSJXXFIVRSETD7IGNYZPWNRQLBZ3DLGO532SNFEFIQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc, address:104.36.82.44:4898): protocol: rpc: dial tcp 104.36.82.44:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048846403 (190h28m48.479926534s) [noise=true , address=104.36.82.35:4602 , piece_id=RJ7ELTYV7AEHVCO7YHNFLNPYWVVEXPDSBB3TTXF4RITPKJQQ54KQ, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:12oPZgRG4QqgnQLo1fgyGYoX1zd34wsd5n7p7L3mRbPD8B4JUuB, address:104.36.82.35:4602): protocol: write tcp 188.166.225.170:35936->104.36.82.35:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048866611 (190h28m48.479946752s) [noise=true , address=104.36.82.59:4318 , piece_id=NOVBFHNY2JIBGX6VGJOOEWCENYTMXV5E4GSHPSEJLJOGGVIDMYXA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:12abkQbak5f8FneYvCZo5FVi4RNZZfijQZFDd4Y1aacLUNgLnyM, address:104.36.82.59:4318): protocol: write tcp 188.166.225.170:59544->104.36.82.59:4318: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048894555 (190h28m48.479974686s) [noise=true , address=104.36.82.60:4370 , piece_id=2CZ7MKJUK5LRARVLF4Z2DD4J6L4BJFOCOGAFNZ6A3PYPZACEXMWA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC, address:104.36.82.60:4370): protocol: rpc: dial tcp 104.36.82.60:4370: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048910716 (190h28m48.479990837s) [noise=true , address=104.36.82.36:4623 , piece_id=SD7MY6YZ6XO7QYL2H2GUXCHKOTP7Z73E2GROTBXEQDA3YJZVDZIA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5, address:104.36.82.36:4623): protocol: rpc: dial tcp 104.36.82.36:4623: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048957285 (190h28m48.480037406s) [noise=true , address=104.36.82.3:4788 , piece_id=YUHE4S5XCSCSYFQ2D5ACYHVJRIPXVAV4RO6PHZOWRZWC6N7J2LLA, seg_pos={0 0}, upload=198147]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: write tcp 188.166.225.170:49422->104.36.82.3:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:54.048980629 (190h28m48.48006076s) [seg_pos={0 0}, upload=198147]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:54:54.283423811 (190h28m48.714503952s) [upload=198147]: Done uploading segments... Aug 16 15:54:54.283450702 (190h28m48.714530823s) [upload=198147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236236829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....................X......X..XXXXXXXXXX.XX.XXXXXX......X...] duration: 6.832482939s FAIL (2025-08-16 15:54:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.832503599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [.....X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236776401s FAIL (2025-08-16 15:54:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Aug 16 15:53:34.826060958 (190h27m29.257141119s) [upload=198141]: Starting upload Aug 16 15:53:34.826155880 (190h27m29.257236011s) [upload=198141]: Done starting upload Aug 16 15:53:34.826263456 (190h27m29.257343637s) [upload=198141]: Uploading segments... Aug 16 15:53:34.826584882 (190h27m29.257665053s) [upload=198141]: Got next segment. Inline: false Aug 16 15:53:34.826639216 (190h27m29.257719387s) [upload=198141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:53:34.826647431 (190h27m29.257727592s) [upload=198141]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:53:35.277686245 (190h27m29.708766376s) [seg_pos={0 0}, upload=198141]: Begin upload segment... Aug 16 15:53:35.277751991 (190h27m29.708832122s) [noise=true , address=77.90.31.21:4890 , piece_id=NKOHA4B2P6XJHLAICDC3GVAZKNRATKIS3ZADOI6E466RPVFRQRIQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278168969 (190h27m29.7092491s) [noise=true , address=45.91.116.13:4187 , piece_id=6NC7Z56J7BRHIQ3KWOK4EUNSCPI7LKZK4HPDCLHNN6PLLJGVH2MQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278309868 (190h27m29.709389999s) [seg_pos={0 0}, upload=198141]: Done begin upload segment. Aug 16 15:53:35.278390262 (190h27m29.709470393s) [noise=true , address=104.36.82.60:4361 , piece_id=SH6YCS6H5OFMNFNNOQVFQJQPNB3X35YKFHPLGFAR7QDTDRLP5OUA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278433825 (190h27m29.709513956s) [noise=true , address=79.127.201.214:4309 , piece_id=KMMR365SSXPZY34M2AGYBGUNQUA2SGAIVNDWIB3TSDBFEVCKHSTA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278696378 (190h27m29.709776509s) [noise=true , address=104.36.82.36:4622 , piece_id=UJSKAFTDL72ZESMM6Z3673VIVKUSCF2HRNRY7TM4H4ITGGQDWNFA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278843619 (190h27m29.70992375s) [noise=true , address=104.36.82.35:4597 , piece_id=7467R54DFEY66BLAZVIZQVQRJ3DG66SEDIBKO44UBLDHX7EVWRMA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.278952558 (190h27m29.710032679s) [noise=true , address=45.91.116.10:4105 , piece_id=QS3TIWFDGF7U7BNWEQGOHZXHGUBNZMWUI4PNFITCNHBQPQBIQ4JA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.279158823 (190h27m29.710238944s) [noise=true , address=77.90.31.11:4575 , piece_id=J53YMZWKMWVUPVCT7Q2G6IGU3ZQFGL3I6AFJDAVL3WFJPHZ2N4NA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.279207226 (190h27m29.710287357s) [noise=true , address=77.90.31.13:4645 , piece_id=4JAYVP3OXCHKPXCNVUD2VLBV5RTXI7IJ6AG2A5TJSTV35PSVBTFQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.279473875 (190h27m29.710553996s) [noise=true , address=104.36.82.44:4896 , piece_id=JPTBZ2R2ZHGNBWQXEEK65HQECWYEY7WAK22LISLA7SM2N4V42WCQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.279763430 (190h27m29.710843551s) [noise=true , address=45.91.116.5:4300 , piece_id=MROK2ES7SFXDJLSRLZU7JEG6ZCR2ZJ5C3PU5CXRT7LMQCT5WSVFQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.280022335 (190h27m29.711102466s) [noise=true , address=104.36.82.58:4266 , piece_id=IF2JDD5MHOIOSC3NRDNZUNXUPV57OYOZCVUIRAYWH33DAGX3U7GQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.280314283 (190h27m29.711394415s) [noise=true , address=104.36.82.59:4306 , piece_id=ZIKKXDLWLEQEQN4FMBJEDLQOLR3CJ2UOGOEH57ZR6IPMCZGTHTYA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.280569251 (190h27m29.711649372s) [noise=true , address=77.90.31.19:4852 , piece_id=TZLHFUIEE5CGBP4O4LE6AF3PNMMQW2L2J3MVOZY6XRMZPC5RPYSQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.280845069 (190h27m29.7119252s) [noise=true , address=79.127.201.215:4344 , piece_id=F5DNTBUAT4HK3CSWXE3IMVD62RJZJDHN3BVSG4EVWYN4JBNZEAYA, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.281083004 (190h27m29.712163135s) [noise=true , address=66.23.193.175:4555 , piece_id=TTCLWJHXDXEGJAXKR2IQNPK4A2JAM5ZUYEBVLET5YRBCV6WKA4UQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.281308556 (190h27m29.712388687s) [noise=true , address=104.36.82.23:4446 , piece_id=7NUWFM7XPWJ64GRQVXQCJPL5KVPWBRUENPWLEN75N5LYMFULXBKQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:35.281532354 (190h27m29.712612485s) [noise=true , address=77.90.31.12:4607 , piece_id=BPUJEQZQGHVRZ3IV5VS2MWYZAFJWXJY6WEKGZLIFO3KCKWLXHTXQ, seg_pos={0 0}, upload=198141]: Uploading piece... Aug 16 15:53:36.751700330 (190h27m31.182780461s) [upload=198141]: Next returned nil segment Aug 16 15:53:36.751718184 (190h27m31.182798305s) [upload=198141]: Waiting for error group managing segments... Aug 16 15:53:37.308584026 (190h27m31.739664147s) [noise=true , address=77.90.31.11:4575 , piece_id=J53YMZWKMWVUPVCT7Q2G6IGU3ZQFGL3I6AFJDAVL3WFJPHZ2N4NA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:37.331563806 (190h27m31.762643937s) [noise=true , address=79.127.201.214:4309 , piece_id=KMMR365SSXPZY34M2AGYBGUNQUA2SGAIVNDWIB3TSDBFEVCKHSTA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:37.472949161 (190h27m31.904029312s) [noise=true , address=45.91.116.13:4187 , piece_id=6NC7Z56J7BRHIQ3KWOK4EUNSCPI7LKZK4HPDCLHNN6PLLJGVH2MQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:37.501026664 (190h27m31.932106795s) [noise=true , address=77.90.31.12:4607 , piece_id=BPUJEQZQGHVRZ3IV5VS2MWYZAFJWXJY6WEKGZLIFO3KCKWLXHTXQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:37.876409227 (190h27m32.307489358s) [noise=true , address=45.91.116.10:4105 , piece_id=QS3TIWFDGF7U7BNWEQGOHZXHGUBNZMWUI4PNFITCNHBQPQBIQ4JA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:40.619842117 (190h27m35.050922238s) [noise=true , address=45.91.116.5:4300 , piece_id=MROK2ES7SFXDJLSRLZU7JEG6ZCR2ZJ5C3PU5CXRT7LMQCT5WSVFQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:43.429573962 (190h27m37.860654093s) [noise=true , address=79.127.201.215:4344 , piece_id=F5DNTBUAT4HK3CSWXE3IMVD62RJZJDHN3BVSG4EVWYN4JBNZEAYA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:45.680126058 (190h27m40.111206189s) [noise=true , address=66.23.193.175:4555 , piece_id=TTCLWJHXDXEGJAXKR2IQNPK4A2JAM5ZUYEBVLET5YRBCV6WKA4UQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:46.817675534 (190h27m41.248755665s) [noise=true , address=77.90.31.13:4645 , piece_id=4JAYVP3OXCHKPXCNVUD2VLBV5RTXI7IJ6AG2A5TJSTV35PSVBTFQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:53:59.830325375 (190h27m54.261405506s) [noise=true , address=77.90.31.19:4852 , piece_id=TZLHFUIEE5CGBP4O4LE6AF3PNMMQW2L2J3MVOZY6XRMZPC5RPYSQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: <nil> Aug 16 15:54:34.179825585 (190h28m28.610905706s) [noise=true , address=77.90.31.21:4890 , piece_id=NKOHA4B2P6XJHLAICDC3GVAZKNRATKIS3ZADOI6E466RPVFRQRIQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: rpc: dial tcp 77.90.31.21:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.179869038 (190h28m28.610949169s) [noise=true , address=104.36.82.35:4597 , piece_id=7467R54DFEY66BLAZVIZQVQRJ3DG66SEDIBKO44UBLDHX7EVWRMA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:12GsSshFB2sM59Asac3cP8EbaTXGZLUUQYTBDxWzPpfiRipQfsf, address:104.36.82.35:4597): protocol: rpc: dial tcp 104.36.82.35:4597: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.179899135 (190h28m28.610979256s) [noise=true , address=104.36.82.36:4622 , piece_id=UJSKAFTDL72ZESMM6Z3673VIVKUSCF2HRNRY7TM4H4ITGGQDWNFA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:104.36.82.36:4622): protocol: rpc: dial tcp 104.36.82.36:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.179926798 (190h28m28.611006929s) [noise=true , address=104.36.82.44:4896 , piece_id=JPTBZ2R2ZHGNBWQXEEK65HQECWYEY7WAK22LISLA7SM2N4V42WCQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: rpc: dial tcp 104.36.82.44:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.179953930 (190h28m28.611034061s) [noise=true , address=104.36.82.58:4266 , piece_id=IF2JDD5MHOIOSC3NRDNZUNXUPV57OYOZCVUIRAYWH33DAGX3U7GQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:104.36.82.58:4266): protocol: rpc: dial tcp 104.36.82.58:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.179983827 (190h28m28.611063958s) [noise=true , address=104.36.82.59:4306 , piece_id=ZIKKXDLWLEQEQN4FMBJEDLQOLR3CJ2UOGOEH57ZR6IPMCZGTHTYA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:104.36.82.59:4306): protocol: rpc: dial tcp 104.36.82.59:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.180010117 (190h28m28.611090238s) [noise=true , address=104.36.82.23:4446 , piece_id=7NUWFM7XPWJ64GRQVXQCJPL5KVPWBRUENPWLEN75N5LYMFULXBKQ, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, address:104.36.82.23:4446): protocol: rpc: dial tcp 104.36.82.23:4446: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.180038912 (190h28m28.611119043s) [noise=true , address=104.36.82.60:4361 , piece_id=SH6YCS6H5OFMNFNNOQVFQJQPNB3X35YKFHPLGFAR7QDTDRLP5OUA, seg_pos={0 0}, upload=198141]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: write tcp 188.166.225.170:47114->104.36.82.60:4361: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:54:34.180051808 (190h28m28.611131949s) [seg_pos={0 0}, upload=198141]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 16 15:54:34.414171789 (190h28m28.84525192s) [upload=198141]: Done uploading segments... Aug 16 15:54:34.414197899 (190h28m28.84527803s) [upload=198141]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.236785889s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.144940149s FAIL (2025-08-16 15:54:31 UTC) orbiter warning: context deadline exceeded Duration: 6.144960018s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X..X.....X......................X...........] duration: 2.223309999s FAIL (2025-08-16 15:53:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.223339746s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228100744s FAIL (2025-08-16 15:53:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== LM6BFAPTFL5Z4M7QIARAQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685119579 (190h26m49.1161997s) [noise=true , address=77.90.31.14:4667 , piece_id=JH2GQTK3RVFMAA4WUZEISMDYUFYJOWCCJXP5WAXLNC2SUQ6OA7MQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685359889 (190h26m49.11644002s) [noise=true , address=104.36.82.23:4475 , piece_id=GZT56G5ZXDIRETKGU3HDM6LNO5RW7GX3NCU6WSMCQKULFG45HDKQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685417189 (190h26m49.11649732s) [noise=true , address=104.36.82.44:4878 , piece_id=KAH7XHFVR4RKOQNQQGQOVOHBQUZOTEQS4226HXNUQ7JIW6JCGJVA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685641638 (190h26m49.116721769s) [noise=true , address=45.91.116.8:4058 , piece_id=GFVEC3V5SCHYF6YZWOHKXTB55W4FTNVVHHNJXHJYFY23WGDHSJDQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685666225 (190h26m49.116746356s) [noise=true , address=79.127.201.215:4354 , piece_id=EKY2XSJDBRRP3MCMXIS3KFS6JNWIAINBSMKQHMQHS4ARW7XXBEFA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.685967782 (190h26m49.117047903s) [noise=true , address=104.36.82.34:4571 , piece_id=WV4X2RWT3UGL5TC2A5SXIQ5KUN7YQJAJCQVEBNO4J44AGI6R2UBA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.686247156 (190h26m49.117327277s) [noise=true , address=104.36.82.38:4712 , piece_id=W75VTJYDAU3H7DAIT7NIIALLFQNCODHMYXHH3C2LA4JU5OASMCXA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.686483469 (190h26m49.1175636s) [noise=true , address=104.36.82.58:4278 , piece_id=FSW6JNWMIIG4HMGMMUKJ4LM3CGKIHIVGKYCUX44VJK3OMQTOMLLQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.686698359 (190h26m49.11777849s) [noise=true , address=104.36.82.50:4037 , piece_id=Y5OQRAJIH26PUMQ5CDFIPKG353CHSZVALTGUH4EEERQ3H5ZFASKA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.686938208 (190h26m49.118018329s) [noise=true , address=77.90.31.12:4595 , piece_id=3IA3ANIW6V77RFUTIWYE7XA62RY4C3BYI4BD5HJGRQNGEQEFQIPA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.687172076 (190h26m49.118252207s) [noise=true , address=77.90.31.15:4725 , piece_id=WVUO2G53PT7UFYENDPUJBMXQU4YK2EA7KM7EHW3WM3NKVZ2IWYBQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.687405312 (190h26m49.118485433s) [noise=true , address=104.36.82.5:4859 , piece_id=3TXGDB3VVLH2YPFSRXOD773CVRSFHSMVAGVDFLTS7ADZAHG3245A, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.687614553 (190h26m49.118694684s) [noise=true , address=45.91.116.5:4326 , piece_id=SOXK7HTJSF2ULDIB666L6QXFENUKJ7WFPMK5Z4B4H7JIGAWU6EPA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.687819975 (190h26m49.118900107s) [noise=true , address=77.90.31.11:4567 , piece_id=7MQXTUFJ5D44BEA6AO5XJBV2CJVVOUPZOHIPJKWAQ3D32JRFB5SQ, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:54.688089612 (190h26m49.119169743s) [noise=true , address=77.90.31.13:4625 , piece_id=A3J5EKEP6BLQ4QR2IYCKMKFCUSJRMQCLTPI5Z77NOD2YJR7NEDSA, seg_pos={0 0}, upload=198129]: Uploading piece... Aug 16 15:52:55.848441798 (190h26m50.279521919s) [upload=198129]: Next returned nil segment Aug 16 15:52:55.848460464 (190h26m50.279540595s) [upload=198129]: Waiting for error group managing segments... Aug 16 15:52:56.061457591 (190h26m50.492537732s) [noise=true , address=104.36.82.50:4037 , piece_id=Y5OQRAJIH26PUMQ5CDFIPKG353CHSZVALTGUH4EEERQ3H5ZFASKA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: storage node overloaded, request limit: 2000 Aug 16 15:52:56.144655190 (190h26m50.575735311s) [noise=true , address=66.23.193.175:4553 , piece_id=JALGWGGUDH5Z4S5OUH2KRWTZUL54XSETNXRREVQAAKMBQC3TG2GQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:56.630536481 (190h26m51.061616612s) [noise=true , address=45.91.116.5:4326 , piece_id=SOXK7HTJSF2ULDIB666L6QXFENUKJ7WFPMK5Z4B4H7JIGAWU6EPA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:56.645514569 (190h26m51.0765947s) [noise=true , address=77.90.31.11:4567 , piece_id=7MQXTUFJ5D44BEA6AO5XJBV2CJVVOUPZOHIPJKWAQ3D32JRFB5SQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:56.698441309 (190h26m51.12952144s) [noise=true , address=77.90.31.12:4595 , piece_id=3IA3ANIW6V77RFUTIWYE7XA62RY4C3BYI4BD5HJGRQNGEQEFQIPA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:56.769323013 (190h26m51.200403144s) [noise=true , address=77.90.31.15:4725 , piece_id=WVUO2G53PT7UFYENDPUJBMXQU4YK2EA7KM7EHW3WM3NKVZ2IWYBQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:56.773514472 (190h26m51.204594604s) [noise=true , address=45.91.116.8:4058 , piece_id=GFVEC3V5SCHYF6YZWOHKXTB55W4FTNVVHHNJXHJYFY23WGDHSJDQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:52:57.797413410 (190h26m52.228493541s) [noise=true , address=77.90.31.14:4667 , piece_id=JH2GQTK3RVFMAA4WUZEISMDYUFYJOWCCJXP5WAXLNC2SUQ6OA7MQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:53:06.400937774 (190h27m0.832017905s) [noise=true , address=79.127.201.215:4354 , piece_id=EKY2XSJDBRRP3MCMXIS3KFS6JNWIAINBSMKQHMQHS4ARW7XXBEFA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:53:46.125426392 (190h27m40.556506523s) [noise=true , address=77.90.31.13:4625 , piece_id=A3J5EKEP6BLQ4QR2IYCKMKFCUSJRMQCLTPI5Z77NOD2YJR7NEDSA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: <nil> Aug 16 15:53:53.597752788 (190h27m48.028832919s) [noise=true , address=104.36.82.59:4305 , piece_id=CLCM3OKI6WUCGVTR67GNGTNAFLGLFYSLM6BFAPTFL5Z4M7QIARAQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:122nFQhbiCN9A3p3qQFsWKwXFFcmtWRXDLBAP57rcLcQBVhShHB, address:104.36.82.59:4305): protocol: write tcp 188.166.225.170:58532->104.36.82.59:4305: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.597828854 (190h27m48.028908985s) [noise=true , address=104.36.82.5:4859 , piece_id=3TXGDB3VVLH2YPFSRXOD773CVRSFHSMVAGVDFLTS7ADZAHG3245A, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: write tcp 188.166.225.170:53696->104.36.82.5:4859: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.597911352 (190h27m48.028991483s) [noise=true , address=104.36.82.23:4475 , piece_id=GZT56G5ZXDIRETKGU3HDM6LNO5RW7GX3NCU6WSMCQKULFG45HDKQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:104.36.82.23:4475): protocol: write tcp 188.166.225.170:36488->104.36.82.23:4475: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.597973020 (190h27m48.029053151s) [noise=true , address=104.36.82.8:4988 , piece_id=LVN5YHKRBU4NC2F2KJHNNERU3WTX76KC2OTJJ5C5AY355ETDUYCA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:12Qs9EDisetfkwcMS8r1d1twWZoTzaz2QZAo2aJakM53ZyJs2Je, address:104.36.82.8:4988): protocol: write tcp 188.166.225.170:48316->104.36.82.8:4988: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.598060427 (190h27m48.029140548s) [noise=true , address=104.36.82.38:4712 , piece_id=W75VTJYDAU3H7DAIT7NIIALLFQNCODHMYXHH3C2LA4JU5OASMCXA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: write tcp 188.166.225.170:55222->104.36.82.38:4712: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.598102337 (190h27m48.029182458s) [noise=true , address=104.36.82.44:4878 , piece_id=KAH7XHFVR4RKOQNQQGQOVOHBQUZOTEQS4226HXNUQ7JIW6JCGJVA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:1wX9xRmyv3KWVyJhFmgBt9uYqDU6U768dqHagKZeKBSEnFup5y, address:104.36.82.44:4878): protocol: write tcp 188.166.225.170:42594->104.36.82.44:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.598168564 (190h27m48.029248695s) [noise=true , address=104.36.82.34:4571 , piece_id=WV4X2RWT3UGL5TC2A5SXIQ5KUN7YQJAJCQVEBNO4J44AGI6R2UBA, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:12v8hKpsCUUAYAKXCs2guky3qeDUd8R2KqjiAo5vCFUATA1WDuW, address:104.36.82.34:4571): protocol: write tcp 188.166.225.170:50486->104.36.82.34:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.598178664 (190h27m48.029258795s) [noise=true , address=104.36.82.58:4278 , piece_id=FSW6JNWMIIG4HMGMMUKJ4LM3CGKIHIVGKYCUX44VJK3OMQTOMLLQ, seg_pos={0 0}, upload=198129]: Done uploading piece. err: ecclient: upload failed (node:12A56U9nehknX9qwMC9j4h2AjfAvEGbjwFFt648E7haU3jusW76, address:104.36.82.58:4278): protocol: write tcp 188.166.225.170:57680->104.36.82.58:4278: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:53.598206086 (190h27m48.029286217s) [seg_pos={0 0}, upload=198129]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:53:53.824017121 (190h27m48.255097242s) [upload=198129]: Done uploading segments... Aug 16 15:53:53.824044984 (190h27m48.255125115s) [upload=198129]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.228113608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.258256687s FAIL (2025-08-16 15:53:53 UTC) orbiter warning: context deadline exceeded Duration: 6.258278507s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....................X......X..XXXXXXXXXX.XX.XXXXXX......X..] duration: 5.891991644s FAIL (2025-08-16 15:53:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.892013806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 36.21224317s FAIL (2025-08-16 15:53:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.212268248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234426405s FAIL (2025-08-16 15:53:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:52:34.369378201 (190h26m28.800458332s) [upload=198124]: Starting upload Aug 16 15:52:34.369454527 (190h26m28.800534659s) [upload=198124]: Done starting upload Aug 16 15:52:34.369527126 (190h26m28.800607257s) [upload=198124]: Uploading segments... Aug 16 15:52:34.369705728 (190h26m28.800785859s) [upload=198124]: Got next segment. Inline: false Aug 16 15:52:34.369736507 (190h26m28.800816638s) [upload=198124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:34.369752939 (190h26m28.80083307s) [upload=198124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:34.878844870 (190h26m29.309925001s) [seg_pos={0 0}, upload=198124]: Begin upload segment... Aug 16 15:52:34.878902180 (190h26m29.309982311s) [seg_pos={0 0}, upload=198124]: Done begin upload segment. Aug 16 15:52:34.878923601 (190h26m29.310003732s) [noise=true , address=104.36.82.23:4450 , piece_id=FR43D33WYQJKWTW6KCMXFODE6GTMALQA7KCBT5YTCGJMJVBD3EYA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.878927779 (190h26m29.3100079s) [noise=true , address=79.127.205.246:4259 , piece_id=RVJ7Z3UUVSHNN3B3N67ELTKH6HTW4GWLIQVNU4GWDJTGCZYP2GDQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.878983976 (190h26m29.310064107s) [noise=true , address=77.90.31.11:4561 , piece_id=G6EMZND37GD6TQKURXU6WPBF4RCKECKIKEFJYCKGN32A65JNYKWQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.879397527 (190h26m29.310477668s) [noise=true , address=79.127.201.214:4322 , piece_id=3OWNONQW7XOVAI6OLXVJWOBIK5S2PMLGLW3UPV6JQXE2GS5HVBKA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.879793334 (190h26m29.310873465s) [noise=true , address=77.90.31.19:4862 , piece_id=G5DKKRXLRU4KRICM4SGFM4QXYGVB33WQ2ZNDSJVPQOPLRL25R5NA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.880398944 (190h26m29.311479075s) [noise=true , address=77.90.31.21:4904 , piece_id=TY7BQ7563NBAKYL7APWR3SLXR4LRDDAQNPNGVKRMWSBXBPYGFEUA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.880670062 (190h26m29.311750193s) [noise=true , address=104.36.82.50:4046 , piece_id=N4OOHCM6JPKHGU4H4BJAMGEVV7OERE7ZA54KWCL4AXKF4PLJGOVA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.880732522 (190h26m29.311812653s) [noise=true , address=45.91.116.5:4305 , piece_id=GIWWAWTP4DFCNIPCQASDXVUYRYPFUHXEF3WYC6MMPPI5ZMI373YA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.880940489 (190h26m29.31202062s) [noise=true , address=104.36.82.34:4561 , piece_id=AD4EMPTYIRFOXWPCML4YMBGAMBNCNW6DCYRADX2CTO5C3SHGMGOA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.881006376 (190h26m29.312086507s) [noise=true , address=77.90.31.14:4696 , piece_id=APAUVLT325DCQBAZR6B5LBWMKG57DPKN6U5UOOFL6DKG2MIEPWOQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.881215466 (190h26m29.312295597s) [noise=true , address=104.36.82.60:4349 , piece_id=XGCALKCWBI5P5P3MVVEEJAVTRB4EJZKRNUAUHVAWB55EBAQOSIOQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.881470755 (190h26m29.312550886s) [noise=true , address=104.36.82.38:4714 , piece_id=Z4QHTEO5QTZEAJ4SEXDRIYBWODTO46AEEOJUIQC5N5O5CKYU7FXA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.881698951 (190h26m29.312779082s) [noise=true , address=104.36.82.59:4322 , piece_id=RPGMQFSX7TF2QITJ5ULCSHC724PNDBTFWXLBPBXMGHQMH5FF5VRQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.881948689 (190h26m29.31302882s) [noise=true , address=104.36.82.5:4842 , piece_id=7XYCK464KCDLFQMQYVWMV5GZRWZACSJDT3WPKY5MBO2NHHLB42HA, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.882175102 (190h26m29.313255233s) [noise=true , address=77.90.31.18:4819 , piece_id=CE2GD7SU7GY547OOWEUR53ML3XDJFZ7S5VTBGYK2JQ7TAE53G2AQ, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.882397838 (190h26m29.313477969s) [noise=true , address=104.36.82.44:4886 , piece_id=6DXQTKDUZ43JOPMDTJDPMGIVFBV4PD77JFXQVDETR7FPAVU4YA5Q, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.882620435 (190h26m29.313700556s) [noise=true , address=79.127.220.101:4070 , piece_id=OYMR56GFTTKVCFQDVJT4ZH24NTSLJHK7Z67KMOQT4FOKAUE4RJ3A, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:34.882835084 (190h26m29.313915215s) [noise=true , address=77.90.31.12:4606 , piece_id=2X3YUUUSZ2AGGJDH2DKHR7IX3HHKVI6AYUJ53EKPB2BBIQO47J6A, seg_pos={0 0}, upload=198124]: Uploading piece... Aug 16 15:52:36.012419172 (190h26m30.443499293s) [upload=198124]: Next returned nil segment Aug 16 15:52:36.012438719 (190h26m30.44351885s) [upload=198124]: Waiting for error group managing segments... Aug 16 15:52:36.272724858 (190h26m30.703804989s) [noise=true , address=104.36.82.50:4046 , piece_id=N4OOHCM6JPKHGU4H4BJAMGEVV7OERE7ZA54KWCL4AXKF4PLJGOVA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12EsoehnMdbRs4cRP66HD93ZWZuzpMWYkfo1VxvQLyQcodAEm7C, address:104.36.82.50:4046): protocol: storage node overloaded, request limit: 2000 Aug 16 15:52:36.301822403 (190h26m30.732902534s) [noise=true , address=79.127.205.246:4259 , piece_id=RVJ7Z3UUVSHNN3B3N67ELTKH6HTW4GWLIQVNU4GWDJTGCZYP2GDQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:36.890833481 (190h26m31.321913612s) [noise=true , address=79.127.201.214:4322 , piece_id=3OWNONQW7XOVAI6OLXVJWOBIK5S2PMLGLW3UPV6JQXE2GS5HVBKA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:36.955669438 (190h26m31.386749569s) [noise=true , address=77.90.31.14:4696 , piece_id=APAUVLT325DCQBAZR6B5LBWMKG57DPKN6U5UOOFL6DKG2MIEPWOQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:37.428244268 (190h26m31.859324399s) [noise=true , address=77.90.31.11:4561 , piece_id=G6EMZND37GD6TQKURXU6WPBF4RCKECKIKEFJYCKGN32A65JNYKWQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:38.380022480 (190h26m32.811102611s) [noise=true , address=45.91.116.5:4305 , piece_id=GIWWAWTP4DFCNIPCQASDXVUYRYPFUHXEF3WYC6MMPPI5ZMI373YA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:42.413129718 (190h26m36.844209849s) [noise=true , address=79.127.220.101:4070 , piece_id=OYMR56GFTTKVCFQDVJT4ZH24NTSLJHK7Z67KMOQT4FOKAUE4RJ3A, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:52:52.913722445 (190h26m47.344802576s) [noise=true , address=77.90.31.12:4606 , piece_id=2X3YUUUSZ2AGGJDH2DKHR7IX3HHKVI6AYUJ53EKPB2BBIQO47J6A, seg_pos={0 0}, upload=198124]: Done uploading piece. err: <nil> Aug 16 15:53:33.715346459 (190h27m28.14642659s) [noise=true , address=104.36.82.59:4322 , piece_id=RPGMQFSX7TF2QITJ5ULCSHC724PNDBTFWXLBPBXMGHQMH5FF5VRQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12bELQDDH69yamGywE32K8YhpPbMRVorkpm85ZFuAtb2CD58ZPH, address:104.36.82.59:4322): protocol: rpc: dial tcp 104.36.82.59:4322: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715360024 (190h27m28.146440155s) [noise=true , address=104.36.82.44:4886 , piece_id=6DXQTKDUZ43JOPMDTJDPMGIVFBV4PD77JFXQVDETR7FPAVU4YA5Q, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715392276 (190h27m28.146472407s) [noise=true , address=104.36.82.38:4714 , piece_id=Z4QHTEO5QTZEAJ4SEXDRIYBWODTO46AEEOJUIQC5N5O5CKYU7FXA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12CkzMG18TDd4EErV26gKMzQ3RAqU4GGnYUqknpRomyuqEk7Emy, address:104.36.82.38:4714): protocol: rpc: dial tcp 104.36.82.38:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715419237 (190h27m28.146499358s) [noise=true , address=104.36.82.23:4450 , piece_id=FR43D33WYQJKWTW6KCMXFODE6GTMALQA7KCBT5YTCGJMJVBD3EYA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms, address:104.36.82.23:4450): protocol: write tcp 188.166.225.170:48186->104.36.82.23:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715426131 (190h27m28.146506262s) [noise=true , address=104.36.82.34:4561 , piece_id=AD4EMPTYIRFOXWPCML4YMBGAMBNCNW6DCYRADX2CTO5C3SHGMGOA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12NTJdWYbLU6o4MTfyp7iPzJCmWYG288BgdGSN8JDcLyYGhCSkb, address:104.36.82.34:4561): protocol: rpc: dial tcp 104.36.82.34:4561: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715456790 (190h27m28.146536921s) [noise=true , address=104.36.82.60:4349 , piece_id=XGCALKCWBI5P5P3MVVEEJAVTRB4EJZKRNUAUHVAWB55EBAQOSIOQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:104.36.82.60:4349): protocol: rpc: dial tcp 104.36.82.60:4349: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715465847 (190h27m28.146545978s) [noise=true , address=77.90.31.21:4904 , piece_id=TY7BQ7563NBAKYL7APWR3SLXR4LRDDAQNPNGVKRMWSBXBPYGFEUA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: rpc: dial tcp 77.90.31.21:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715485555 (190h27m28.146565686s) [noise=true , address=104.36.82.5:4842 , piece_id=7XYCK464KCDLFQMQYVWMV5GZRWZACSJDT3WPKY5MBO2NHHLB42HA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715498599 (190h27m28.14657873s) [noise=true , address=77.90.31.19:4862 , piece_id=G5DKKRXLRU4KRICM4SGFM4QXYGVB33WQ2ZNDSJVPQOPLRL25R5NA, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 188.166.225.170:55114->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715519169 (190h27m28.1465993s) [noise=true , address=77.90.31.18:4819 , piece_id=CE2GD7SU7GY547OOWEUR53ML3XDJFZ7S5VTBGYK2JQ7TAE53G2AQ, seg_pos={0 0}, upload=198124]: Done uploading piece. err: ecclient: upload failed (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da, address:77.90.31.18:4819): protocol: read tcp 188.166.225.170:58408->77.90.31.18:4819: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:33.715542293 (190h27m28.146622424s) [seg_pos={0 0}, upload=198124]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:53:33.947602255 (190h27m28.378682386s) [upload=198124]: Done uploading segments... Aug 16 15:53:33.947627082 (190h27m28.378707213s) [upload=198124]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.234447866s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......................X......X..XXXXXXXXXX.XX.XXXXXX......X.] duration: 9.355876317s FAIL (2025-08-16 15:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.355897057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................X......X..XXXXXXXXXX.XX.XXXXXX......X] duration: 30.230493262s FAIL (2025-08-16 15:53:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:52:48.101266066 (190h26m42.532346197s) [upload=198126]: Starting upload Aug 16 15:52:48.101317784 (190h26m42.532397905s) [upload=198126]: Done starting upload Aug 16 15:52:48.101396355 (190h26m42.532476486s) [upload=198126]: Uploading segments... Aug 16 15:52:48.101555870 (190h26m42.532636001s) [upload=198126]: Got next segment. Inline: false Aug 16 15:52:48.101586769 (190h26m42.53266689s) [upload=198126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:52:48.101591238 (190h26m42.532671369s) [upload=198126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:52:48.334828191 (190h26m42.765908322s) [seg_pos={0 0}, upload=198126]: Begin upload segment... Aug 16 15:52:48.334863479 (190h26m42.76594361s) [seg_pos={0 0}, upload=198126]: Done begin upload segment. Aug 16 15:52:48.334885220 (190h26m42.765965351s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.334893316 (190h26m42.765973447s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335238687 (190h26m42.766318818s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335474287 (190h26m42.766554428s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335487924 (190h26m42.766568055s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335738082 (190h26m42.766818203s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335887598 (190h26m42.766967729s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.335987599 (190h26m42.76706773s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336171752 (190h26m42.767251883s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336204875 (190h26m42.767284996s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336433623 (190h26m42.767513754s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336644196 (190h26m42.767724327s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.336838167 (190h26m42.767918298s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337041085 (190h26m42.768121216s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337237200 (190h26m42.768317331s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:48.337410723 (190h26m42.768490854s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Uploading piece... Aug 16 15:52:50.079518034 (190h26m44.510598165s) [upload=198126]: Next returned nil segment Aug 16 15:52:50.079534346 (190h26m44.510614477s) [upload=198126]: Waiting for error group managing segments... Aug 16 15:52:50.378124896 (190h26m44.809205027s) [noise=true , address=104.36.82.15:4257 , piece_id=O5P2TRMV263Y47S55SKVB4PU3J2L264BMYQVVCRFDA2I2PKGEN6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.386886641 (190h26m44.817966762s) [noise=true , address=104.36.82.7:4909 , piece_id=SS5U5GQNFLH6W7L7TH6KU27ZSP3MX7B6GHXA7ESD64PADSKUXTHA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.411927724 (190h26m44.843007855s) [noise=true , address=77.90.31.16:4735 , piece_id=7IE47CDMJ37FFQ7T3JJH6ULYRM3POYQC6OR2VSKU4UQZEXA47EAA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.423465300 (190h26m44.854545431s) [noise=true , address=45.91.116.10:4082 , piece_id=STQV2TXNXGDKPL6PGQRH5YX7G6ZSSZVOZPKATYO627YPOEE5QKOQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.477208261 (190h26m44.908288392s) [noise=true , address=104.36.82.10:4078 , piece_id=4N577VBUXJNDD4PHWBZBISVEJ5JJOKFYZMZ3CDUXNY7WZMUECQ5A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.482586732 (190h26m44.913666863s) [noise=true , address=104.36.82.10:4079 , piece_id=GNF4DQPDWL4WB5U4KN2MHLSP5JKC2YV7MSVZAV4MTNQ37ITBXG6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.495273928 (190h26m44.926354059s) [noise=true , address=45.91.116.4:4937 , piece_id=CAPF5TAZHI6OFG6UJDRKCGXSJZJUZJBE56AGXTM7BFTBQ6X6ZTDA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.566548704 (190h26m44.997628835s) [noise=true , address=104.36.82.6:4873 , piece_id=S5WP5BFEILQGOQDY5HFKOWPEKFYCHWPQEYCHSIC6JAQ3PSFMSIJA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.633542179 (190h26m45.06462231s) [noise=true , address=104.36.82.12:4151 , piece_id=I3BH6GTDSVL3H5H2AUKH2KUH6UV3H36KECOAMOX7EIA45S2CMFZA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:50.858362709 (190h26m45.28944284s) [noise=true , address=45.91.116.14:4190 , piece_id=CLFKYQRJZGPJZDAF45H2U7OK63GVMUCPS5RSL7GSWJUCCP46FI2A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:52:52.066851539 (190h26m46.49793167s) [noise=true , address=104.36.82.7:4910 , piece_id=MPVULMWLYUMWMEONZSC7XRWKLJQH35L242NEZCNPTKQNTNBVW6ZQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: <nil> Aug 16 15:53:17.449938469 (190h27m11.88101859s) [noise=true , address=104.36.82.2:4729 , piece_id=JBMAAJAQJVNPQ7T7Z7PNX2XGTEUSPWB3B6XM7AZ6P4NED77EW6DQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450009375 (190h27m11.881089506s) [noise=true , address=104.36.82.11:4114 , piece_id=S5LL36D5SFPF4KQPSAHUGZFRBKKHO5E5OFKY2IOM6UHPQPYWYS6Q, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450053389 (190h27m11.88113352s) [noise=true , address=104.36.82.14:4222 , piece_id=CJHGHYKDRQKTCB7VOFAHWCKL7ANGU3ODMT7MKVVCL6B7MUKWJZMA, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450090159 (190h27m11.88117029s) [noise=true , address=104.36.82.4:4803 , piece_id=UKSKIX52A5NTHVW6WJZMNFYSHA7BWSSWQCEUGMTUAKJO7723XZ6A, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:53:17.450179460 (190h27m11.881259591s) [noise=true , address=77.90.31.18:4807 , piece_id=64KXLU675Z7LLLZBKBXC6YFRZAVCFUTUP63UNFJUNM2TKWN7NJMQ, seg_pos={0 0}, upload=198126]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op Aug 16 15:53:17.450234095 (190h27m11.881314226s) [seg_pos={0 0}, upload=198126]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:53:17.677756104 (190h27m12.108836235s) [upload=198126]: Done uploading segments... Aug 16 15:53:17.677812451 (190h27m12.108892573s) [upload=198126]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.230516477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...............X.X..X.....X......................X..........] duration: 2.054592158s FAIL (2025-08-16 15:53:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.054612256s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 30.15507182s FAIL (2025-08-16 15:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.155091809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.370137492s FAIL (2025-08-16 15:52:54 UTC) orbiter warning: context deadline exceeded Duration: 6.370157891s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.243543613s FAIL (2025-08-16 15:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== se=true , address=79.127.201.215:4350 , piece_id=MNHGN5QAAHFOZIX7IZFAS25WRXBQIBPSGZXKS5RPEECRO7YNHTLQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.227967902 (190h25m48.659048033s) [noise=true , address=104.36.82.5:4872 , piece_id=V5AJOXUE2NX2MYF2DOJCSO2ZFANLSXPX64JM3TZLOYC4JMQYI55Q, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.228163736 (190h25m48.659243857s) [noise=true , address=104.36.82.44:4890 , piece_id=JLOALBUKN5YHGMEOZL5KRSSSYGMCDSE2O3XETQHSGCSIL2QI3Y4Q, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.228215285 (190h25m48.659295416s) [noise=true , address=104.36.82.60:4337 , piece_id=WOYZHFHCM4NZE5PFC6JEMYUKX2TT237ZAJMQXDFOLCTTVB7A73VQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.228476013 (190h25m48.659556145s) [noise=true , address=104.36.82.39:4747 , piece_id=F3A7D6ELD3RFKBTYZRJWWPWEOZR7GMKDPZA6P3DR5JVIBCRGB56Q, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.228695383 (190h25m48.659775514s) [noise=true , address=66.23.193.175:4562 , piece_id=P6JPBMF5IFCP2WEW4OWX2HLWP5J4BPNHQXAGARX3EENKUD2HTXBA, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.228907860 (190h25m48.659987991s) [noise=true , address=45.91.116.8:4064 , piece_id=RPJYDFFLRDD5LTD3UYMNBOY5KMQKQMCLGP36BIZBY2O26452ZADQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.229136397 (190h25m48.660216528s) [noise=true , address=104.36.82.11:4140 , piece_id=BL5TRGLBAQV5DTM6GSY4L2NZSA46XB4GQTF4LE5FZ7IH6MB3R5CQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.229333374 (190h25m48.660413505s) [noise=true , address=77.90.31.18:4830 , piece_id=FYG5MPVEW7XW3BSCI44JXMENWYUVBPRTXY55SUZXD36OFUNAUTEA, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.229520061 (190h25m48.660600192s) [noise=true , address=77.90.31.12:4608 , piece_id=I4D44WBMSKWLPWGG4GGOQYQ3LFRU6LSIYIL64B2RUTR2U7YENDAQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.229706749 (190h25m48.66078688s) [noise=true , address=104.36.82.59:4318 , piece_id=XZKZICDYBZ2Y4NWNPRAEEO4QE26TV3EAFZAZWKLUQMLYWAEZJROA, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.229923052 (190h25m48.661003184s) [noise=true , address=77.90.31.10:4532 , piece_id=ULM2QBD35YNBAPJ4L7JAAHQOQXXVRYIF4SQ564EBGSOZUILVLMKQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.230116402 (190h25m48.661196523s) [noise=true , address=77.90.31.19:4869 , piece_id=VPXIPMPVN3AVWQXFAEH6FPLAPGKL766E76NXZYG7GJ22ARFPMFTQ, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.230310975 (190h25m48.661391106s) [noise=true , address=45.91.116.13:4156 , piece_id=FLS5BFYUELL6QRTMTYARAC6KCB4CT7J2KWURDNM55APDH4PEGCAA, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:54.231150101 (190h25m48.662230232s) [noise=true , address=45.91.116.10:4107 , piece_id=JR3L5UIO562T2CODW2MOIZQTLO7AIOGXHIS5GSI3YSIFYHBOGN7A, seg_pos={0 0}, upload=198112]: Uploading piece... Aug 16 15:51:55.394161867 (190h25m49.825241998s) [upload=198112]: Next returned nil segment Aug 16 15:51:55.394177557 (190h25m49.825257688s) [upload=198112]: Waiting for error group managing segments... Aug 16 15:51:56.275375623 (190h25m50.706455754s) [noise=true , address=79.127.201.215:4350 , piece_id=MNHGN5QAAHFOZIX7IZFAS25WRXBQIBPSGZXKS5RPEECRO7YNHTLQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:51:59.634066581 (190h25m54.065146712s) [noise=true , address=45.91.116.13:4156 , piece_id=FLS5BFYUELL6QRTMTYARAC6KCB4CT7J2KWURDNM55APDH4PEGCAA, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:00.417369935 (190h25m54.848450066s) [noise=true , address=45.91.116.10:4107 , piece_id=JR3L5UIO562T2CODW2MOIZQTLO7AIOGXHIS5GSI3YSIFYHBOGN7A, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:01.367364777 (190h25m55.798444908s) [noise=true , address=66.23.193.175:4562 , piece_id=P6JPBMF5IFCP2WEW4OWX2HLWP5J4BPNHQXAGARX3EENKUD2HTXBA, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:03.230935577 (190h25m57.662015708s) [noise=true , address=77.90.31.19:4869 , piece_id=VPXIPMPVN3AVWQXFAEH6FPLAPGKL766E76NXZYG7GJ22ARFPMFTQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:04.689170008 (190h25m59.120250129s) [noise=true , address=77.90.31.10:4532 , piece_id=ULM2QBD35YNBAPJ4L7JAAHQOQXXVRYIF4SQ564EBGSOZUILVLMKQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:05.814716310 (190h26m0.245796451s) [noise=true , address=77.90.31.14:4682 , piece_id=R7K33ZLA66JEDZJEJXTTSDKMMXWFGCPV7MHKI2CVQZJ6AOVGDIFA, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:16.016834722 (190h26m10.447914853s) [noise=true , address=77.90.31.18:4830 , piece_id=FYG5MPVEW7XW3BSCI44JXMENWYUVBPRTXY55SUZXD36OFUNAUTEA, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:21.296357948 (190h26m15.727438079s) [noise=true , address=45.91.116.8:4064 , piece_id=RPJYDFFLRDD5LTD3UYMNBOY5KMQKQMCLGP36BIZBY2O26452ZADQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: <nil> Aug 16 15:52:53.123345335 (190h26m47.554425466s) [noise=true , address=104.36.82.58:4277 , piece_id=FFPZ4OVDQBH7R3NTHQB2X227Y6YFLKCMAIWZ5CMLIPE3TENW2IDQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1ZqfDCS5wuZx9FQRcQqjUj8HBrrYyxxZYVEzK4gJg3pRJMzpum, address:104.36.82.58:4277): protocol: write tcp 188.166.225.170:36158->104.36.82.58:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123373740 (190h26m47.554453871s) [noise=true , address=104.36.82.44:4890 , piece_id=JLOALBUKN5YHGMEOZL5KRSSSYGMCDSE2O3XETQHSGCSIL2QI3Y4Q, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123400070 (190h26m47.554480201s) [noise=true , address=104.36.82.60:4337 , piece_id=WOYZHFHCM4NZE5PFC6JEMYUKX2TT237ZAJMQXDFOLCTTVB7A73VQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, address:104.36.82.60:4337): protocol: write tcp 188.166.225.170:47984->104.36.82.60:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123442471 (190h26m47.554522602s) [noise=true , address=104.36.82.11:4140 , piece_id=BL5TRGLBAQV5DTM6GSY4L2NZSA46XB4GQTF4LE5FZ7IH6MB3R5CQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: rpc: dial tcp 104.36.82.11:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123504078 (190h26m47.554584209s) [noise=true , address=104.36.82.39:4747 , piece_id=F3A7D6ELD3RFKBTYZRJWWPWEOZR7GMKDPZA6P3DR5JVIBCRGB56Q, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1bmbaxrj3X67qtdL5boUEPGBsujt1ggcGcGsJvvGTAMWoQe8Qg, address:104.36.82.39:4747): protocol: rpc: dial tcp 104.36.82.39:4747: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123533765 (190h26m47.554613896s) [noise=true , address=104.36.82.38:4695 , piece_id=2QNGOEXWOUC46HXUX47UOBWC2VIWM2EHHLX3FS6ZWNWXNUJWMFLQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y, address:104.36.82.38:4695): protocol: write tcp 188.166.225.170:34418->104.36.82.38:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123562781 (190h26m47.554642912s) [noise=true , address=104.36.82.5:4872 , piece_id=V5AJOXUE2NX2MYF2DOJCSO2ZFANLSXPX64JM3TZLOYC4JMQYI55Q, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: write tcp 188.166.225.170:34058->104.36.82.5:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123573742 (190h26m47.554653863s) [noise=true , address=77.90.31.12:4608 , piece_id=I4D44WBMSKWLPWGG4GGOQYQ3LFRU6LSIYIL64B2RUTR2U7YENDAQ, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: write tcp 188.166.225.170:48000->77.90.31.12:4608: use of closed network connection; write tcp 188.166.225.170:48000->77.90.31.12:4608: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48000->77.90.31.12:4608: use of closed network connection Aug 16 15:52:53.123587899 (190h26m47.55466803s) [noise=true , address=104.36.82.59:4318 , piece_id=XZKZICDYBZ2Y4NWNPRAEEO4QE26TV3EAFZAZWKLUQMLYWAEZJROA, seg_pos={0 0}, upload=198112]: Done uploading piece. err: ecclient: upload failed (node:12abkQbak5f8FneYvCZo5FVi4RNZZfijQZFDd4Y1aacLUNgLnyM, address:104.36.82.59:4318): protocol: write tcp 188.166.225.170:37918->104.36.82.59:4318: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:53.123605713 (190h26m47.554685844s) [seg_pos={0 0}, upload=198112]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:52:53.364802306 (190h26m47.795882437s) [upload=198112]: Done uploading segments... Aug 16 15:52:53.364829118 (190h26m47.795909249s) [upload=198112]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.243567418s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)