Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.253400614s FAIL (2025-08-16 15:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 16 15:51:33.896854608 (190h25m28.327934739s) [upload=198104]: Starting upload Aug 16 15:51:33.896902750 (190h25m28.327982881s) [upload=198104]: Done starting upload Aug 16 15:51:33.896996479 (190h25m28.32807661s) [upload=198104]: Uploading segments... Aug 16 15:51:33.897073046 (190h25m28.328153177s) [upload=198104]: Got next segment. Inline: false Aug 16 15:51:33.897098845 (190h25m28.328178966s) [upload=198104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:33.897103123 (190h25m28.328183254s) [upload=198104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:34.411035507 (190h25m28.842115638s) [seg_pos={0 0}, upload=198104]: Begin upload segment... Aug 16 15:51:34.411070945 (190h25m28.842151076s) [seg_pos={0 0}, upload=198104]: Done begin upload segment. Aug 16 15:51:34.411088969 (190h25m28.842169101s) [noise=true , address=45.91.116.6:4999 , piece_id=ETS44OJN22Q22FJO2NEUPJDGVQMUOABSJ2TK3G2ZC7LHLZ352J7Q, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.411324861 (190h25m28.842404992s) [noise=true , address=79.127.201.215:4357 , piece_id=CBCFNJSPG2IYVGTLXPL64RNCJHIYLEEMFHOGFG6E4KFVUZTVOYLA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.411399974 (190h25m28.842480105s) [noise=true , address=104.36.82.23:4457 , piece_id=W7SYGORHXHG3F637YMH7QTHKGEHR4NEXDD6A33PXZ6RO72XQUSWA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.411573767 (190h25m28.842653898s) [noise=true , address=79.127.201.214:4319 , piece_id=VTI6SVUMGPRJ6KIP337ZXFGBCK7EJNOKDMBH7IJ75TLX36ITVWXQ, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.411731979 (190h25m28.8428121s) [noise=true , address=104.36.82.36:4647 , piece_id=Z6GRDBWJTB5HTQGFKUGZEQ4FDTFL546R5YDINOE4FDXJFEKYOKCA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.412341376 (190h25m28.843421507s) [noise=true , address=77.90.31.11:4585 , piece_id=RIM3KSIPD5EI5CIHKQ27LD2NJMK2N45TU77HBUJX4XWJXESQGE4A, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.412393696 (190h25m28.843473827s) [noise=true , address=66.23.193.175:4554 , piece_id=UBVR3RMKBSLCGV6OJ6KAOWO6WJEN5QQBBPQN6H7NDYK3ZP2R5PHA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.412663432 (190h25m28.843743563s) [noise=true , address=77.90.31.21:4901 , piece_id=KQLQLUPGGPXXUT4IIVONKSEZAWBEKLBCUWKIQDXGTUJVMN6N554A, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.412862954 (190h25m28.843943095s) [noise=true , address=104.36.82.44:4891 , piece_id=CBKWSHS3P5AWFYLYD4AQ5OO7ZNSF2IDWBWNOADT5DMLO37ZCDDJA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.413044441 (190h25m28.844124562s) [noise=true , address=104.36.82.5:4863 , piece_id=XMTWEP7D4NAY4NLO4HXYJEWVVU3P3PBRUFIXXVUG54ND2HQ7IWAQ, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.413460457 (190h25m28.844540588s) [noise=true , address=64.49.13.155:4922 , piece_id=VCUALGXUARMMZLYCA4RILVF3CCZQNTP6WASGU2BCYPL2CIZRO6QA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.413618740 (190h25m28.844698871s) [noise=true , address=77.90.31.18:4826 , piece_id=QNYDQIWS2SDAUMZW2NEXV227RZATAPNOVAMQZSKWC4MXU3ZCUJYA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.414029516 (190h25m28.845109647s) [noise=true , address=77.90.31.16:4757 , piece_id=WJILCDO2SJ7DXFAD7ARMOYF4IVJSYPMZLFOU43QQLOXZVOXSQOXA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.414251661 (190h25m28.845331792s) [noise=true , address=104.36.82.20:4459 , piece_id=RKYWMT6T3GUGCGUCMRXDV2YNRGQHHNFR3FQKEUOY4JGNI45RCN3A, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.415322710 (190h25m28.846402841s) [noise=true , address=45.91.116.13:4182 , piece_id=62VJ4XOOSCE7AWWJXAPTCVRQDUPFWQ5HSB6DRL6LQNTA6AGGYWAA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.415538563 (190h25m28.846618684s) [noise=true , address=104.36.82.58:4277 , piece_id=JP2BR7HFLS6QRSVN2EB6PDTNW26BJTVNQFTPNKTNJCDVXF4QTDFA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.415611783 (190h25m28.846691915s) [noise=true , address=104.36.82.38:4712 , piece_id=7IDNY7YOBUK4WHLCFB6D3XSLVFCFX5ZS34RYUIH223S67ZGPW22Q, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:34.415996890 (190h25m28.847077021s) [noise=true , address=77.90.31.14:4690 , piece_id=Y5QY7BVIKMAYOWF7ORJQX376Z6HMPEI5QRP2P6Z4V5AQF5JOQ4NA, seg_pos={0 0}, upload=198104]: Uploading piece... Aug 16 15:51:35.827594547 (190h25m30.258674678s) [upload=198104]: Next returned nil segment Aug 16 15:51:35.827611760 (190h25m30.258691891s) [upload=198104]: Waiting for error group managing segments... Aug 16 15:51:37.470902287 (190h25m31.901982408s) [noise=true , address=79.127.201.214:4319 , piece_id=VTI6SVUMGPRJ6KIP337ZXFGBCK7EJNOKDMBH7IJ75TLX36ITVWXQ, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:37.673392930 (190h25m32.104473102s) [noise=true , address=64.49.13.155:4922 , piece_id=VCUALGXUARMMZLYCA4RILVF3CCZQNTP6WASGU2BCYPL2CIZRO6QA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:38.582505799 (190h25m33.01358593s) [noise=true , address=66.23.193.175:4554 , piece_id=UBVR3RMKBSLCGV6OJ6KAOWO6WJEN5QQBBPQN6H7NDYK3ZP2R5PHA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:38.947851729 (190h25m33.37893186s) [noise=true , address=77.90.31.16:4757 , piece_id=WJILCDO2SJ7DXFAD7ARMOYF4IVJSYPMZLFOU43QQLOXZVOXSQOXA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:39.022727424 (190h25m33.453807545s) [noise=true , address=77.90.31.21:4901 , piece_id=KQLQLUPGGPXXUT4IIVONKSEZAWBEKLBCUWKIQDXGTUJVMN6N554A, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:40.662280350 (190h25m35.093360481s) [noise=true , address=77.90.31.11:4585 , piece_id=RIM3KSIPD5EI5CIHKQ27LD2NJMK2N45TU77HBUJX4XWJXESQGE4A, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:41.583559879 (190h25m36.01464001s) [noise=true , address=79.127.201.215:4357 , piece_id=CBCFNJSPG2IYVGTLXPL64RNCJHIYLEEMFHOGFG6E4KFVUZTVOYLA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:43.992839141 (190h25m38.423919272s) [noise=true , address=77.90.31.14:4690 , piece_id=Y5QY7BVIKMAYOWF7ORJQX376Z6HMPEI5QRP2P6Z4V5AQF5JOQ4NA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:51:52.431397189 (190h25m46.86247731s) [noise=true , address=45.91.116.6:4999 , piece_id=ETS44OJN22Q22FJO2NEUPJDGVQMUOABSJ2TK3G2ZC7LHLZ352J7Q, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:52:02.904728345 (190h25m57.335808476s) [noise=true , address=77.90.31.18:4826 , piece_id=QNYDQIWS2SDAUMZW2NEXV227RZATAPNOVAMQZSKWC4MXU3ZCUJYA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:52:18.325127265 (190h26m12.756207396s) [noise=true , address=45.91.116.13:4182 , piece_id=62VJ4XOOSCE7AWWJXAPTCVRQDUPFWQ5HSB6DRL6LQNTA6AGGYWAA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: <nil> Aug 16 15:52:33.247927779 (190h26m27.6790079s) [noise=true , address=104.36.82.23:4457 , piece_id=W7SYGORHXHG3F637YMH7QTHKGEHR4NEXDD6A33PXZ6RO72XQUSWA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:12mpuVB9gpUQ6dNLaMAVaDkgctVSetSbPRxhxU49WhAzycwwNhs, address:104.36.82.23:4457): protocol: rpc: dial tcp 104.36.82.23:4457: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:33.247955262 (190h26m27.679035383s) [noise=true , address=104.36.82.38:4712 , piece_id=7IDNY7YOBUK4WHLCFB6D3XSLVFCFX5ZS34RYUIH223S67ZGPW22Q, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:33.247970981 (190h26m27.679051112s) [noise=true , address=104.36.82.36:4647 , piece_id=Z6GRDBWJTB5HTQGFKUGZEQ4FDTFL546R5YDINOE4FDXJFEKYOKCA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:1arYSsvaPDeLVdppvVEB5VYwRAcJkLfYWXMG7CAHvvY4YTKxpZ, address:104.36.82.36:4647): protocol: rpc: dial tcp 104.36.82.36:4647: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:33.248002251 (190h26m27.679082372s) [noise=true , address=104.36.82.44:4891 , piece_id=CBKWSHS3P5AWFYLYD4AQ5OO7ZNSF2IDWBWNOADT5DMLO37ZCDDJA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:33.248006981 (190h26m27.679087112s) [noise=true , address=104.36.82.20:4459 , piece_id=RKYWMT6T3GUGCGUCMRXDV2YNRGQHHNFR3FQKEUOY4JGNI45RCN3A, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:52:33.248011249 (190h26m27.67909138s) [noise=true , address=104.36.82.58:4277 , piece_id=JP2BR7HFLS6QRSVN2EB6PDTNW26BJTVNQFTPNKTNJCDVXF4QTDFA, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:1ZqfDCS5wuZx9FQRcQqjUj8HBrrYyxxZYVEzK4gJg3pRJMzpum, address:104.36.82.58:4277): protocol: rpc: dial tcp 104.36.82.58:4277: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:52:33.248083376 (190h26m27.679163497s) [noise=true , address=104.36.82.5:4863 , piece_id=XMTWEP7D4NAY4NLO4HXYJEWVVU3P3PBRUFIXXVUG54ND2HQ7IWAQ, seg_pos={0 0}, upload=198104]: Done uploading piece. err: ecclient: upload failed (node:12ApjPajPXNsWuwWvZ778CTxL33xrFi2cebEDBNHhFa52Nm2aqn, address:104.36.82.5:4863): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:52:33.248099758 (190h26m27.679179879s) [seg_pos={0 0}, upload=198104]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 16 15:52:33.499209829 (190h26m27.93028997s) [upload=198104]: Done uploading segments... Aug 16 15:52:33.499242131 (190h26m27.930322282s) [upload=198104]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.253425151s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [................X.X..X.....X......................X.........] duration: 2.299549227s FAIL (2025-08-16 15:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.299573553s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................X......X..XXXXXXXXXX.XX.XXXXXX......] duration: 25.41823915s FAIL (2025-08-16 15:52:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 25.418266184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.................X.X..X.....X......................X........] duration: 2.22737655s FAIL (2025-08-16 15:52:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.227407469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 21.128388183s FAIL (2025-08-16 15:51:58 UTC) orbiter warning: context deadline exceeded Duration: 21.128469037s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 23.43610751s FAIL (2025-08-16 15:51:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.436119072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227171145s FAIL (2025-08-16 15:51:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ... Aug 16 15:50:45.865751247 (190h24m40.296831378s) [noise=true , address=104.36.82.44:4891 , piece_id=CZILE233FG5QMOAHD4H2FDOID4DWUM45QHICCUPORKRACIGNKQJA, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.865996767 (190h24m40.297076898s) [noise=true , address=104.36.82.35:4605 , piece_id=XPU7NC75Q4JG6N3H35AEKWAMMQIXYKLGDCN6ERELB3G3ES3QQN6A, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.866244531 (190h24m40.297324652s) [noise=true , address=104.36.82.39:4749 , piece_id=XBGMDHL6EO5ENX2AIPRCEKIWJX3KNSEDFLEZGKTMYRLXJFIYX33A, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.866471305 (190h24m40.297551436s) [noise=true , address=79.127.201.215:4347 , piece_id=I4TBN3FVCRHPDYKKB7LIXFRNNDBXJMITHWRG76SDHSXJND4EXKGQ, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.866680545 (190h24m40.297760666s) [noise=true , address=104.36.82.36:4622 , piece_id=3ZIUVEUCKS7Z2435GLXGAKLXHCOQSGVRFQX7SZRVLL7JUOXU6AYA, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.866874125 (190h24m40.297954257s) [noise=true , address=104.36.82.2:4748 , piece_id=XPPZWROXUCBPM7GLEFC34ODEXGPXQKQSTEFDVZ5NRNGG2I5XFDXA, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.867336090 (190h24m40.298416221s) [noise=true , address=104.36.82.23:4473 , piece_id=7JX2BE672JZMXSUBWAX4IQGJ5G424DSFNR3F5TSDO2O7YPW6GJFA, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.867777384 (190h24m40.298857515s) [noise=true , address=104.36.82.58:4293 , piece_id=V42QWQHQPDR45TRH7DKEPM3IYHPGTUQGB3TXSO4TXR4WC6CZKWDQ, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.868312517 (190h24m40.299392648s) [noise=true , address=77.90.31.14:4664 , piece_id=4EVKGLM6XQ6KM5PNX6MKQNYC5JM24CFQGS7FSLUGHE74BEDPFY4A, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.868792276 (190h24m40.299872407s) [noise=true , address=66.23.193.175:4564 , piece_id=HRIJKMINUHUQ64UPDS5VCGLOZ5ATGHDHYT4ENHXXV6M2F3U4FJHQ, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.869363148 (190h24m40.300443279s) [noise=true , address=104.36.82.4:4812 , piece_id=GKEQOCQEXDCW3W7NRAGR25GATAKM5TRF55QZFVQSVN4KB5VXGC7Q, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.869763424 (190h24m40.300843555s) [noise=true , address=77.90.31.21:4889 , piece_id=QMABFWGIQHRQPOHBSLZU2YNSHHGCI4CLGGNG5MPLYGAF3ETYBMFA, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.870412366 (190h24m40.301492487s) [noise=true , address=45.91.116.8:4051 , piece_id=JUVAY5MVJ4QF3HEZSLYVB4G54ZBOQIDPCSUFDELBXMVYE3JX5GYQ, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:45.870639109 (190h24m40.30171923s) [noise=true , address=104.36.82.60:4339 , piece_id=SLTM2YLQ7S7K47X4D75O6Q4PVCRLNA73CIPLOESPVMYLJANCERCQ, seg_pos={0 0}, upload=198092]: Uploading piece... Aug 16 15:50:47.367280717 (190h24m41.798360848s) [upload=198092]: Next returned nil segment Aug 16 15:50:47.367300956 (190h24m41.798381087s) [upload=198092]: Waiting for error group managing segments... Aug 16 15:50:49.641339489 (190h24m44.07241962s) [noise=true , address=77.90.31.10:4542 , piece_id=EPG56RPO5XOHZ7WHQFH5BEWONLDU3YYNTX7SLLVI7JC3TRGUUZVQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:50:49.769921027 (190h24m44.201001158s) [noise=true , address=77.90.31.14:4664 , piece_id=4EVKGLM6XQ6KM5PNX6MKQNYC5JM24CFQGS7FSLUGHE74BEDPFY4A, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:50:50.215144463 (190h24m44.646224594s) [noise=true , address=77.90.31.18:4827 , piece_id=DASD56HE5AMRRCQ7I2FFZ5U4TJNXFEOTY7OITKAVSB4S3RSFH4NQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:50:53.403897043 (190h24m47.834977174s) [noise=true , address=45.91.116.8:4051 , piece_id=JUVAY5MVJ4QF3HEZSLYVB4G54ZBOQIDPCSUFDELBXMVYE3JX5GYQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:50:54.212012764 (190h24m48.643092885s) [noise=true , address=66.23.193.175:4564 , piece_id=HRIJKMINUHUQ64UPDS5VCGLOZ5ATGHDHYT4ENHXXV6M2F3U4FJHQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:50:54.379312264 (190h24m48.810392395s) [noise=true , address=79.127.201.215:4347 , piece_id=I4TBN3FVCRHPDYKKB7LIXFRNNDBXJMITHWRG76SDHSXJND4EXKGQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:51:07.566478284 (190h25m1.997558416s) [noise=true , address=77.90.31.21:4889 , piece_id=QMABFWGIQHRQPOHBSLZU2YNSHHGCI4CLGGNG5MPLYGAF3ETYBMFA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: <nil> Aug 16 15:51:44.752849668 (190h25m39.183929799s) [noise=true , address=104.36.82.2:4748 , piece_id=XPPZWROXUCBPM7GLEFC34ODEXGPXQKQSTEFDVZ5NRNGG2I5XFDXA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:104.36.82.2:4748): protocol: rpc: dial tcp 104.36.82.2:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.752899072 (190h25m39.183979203s) [noise=true , address=104.36.82.23:4473 , piece_id=7JX2BE672JZMXSUBWAX4IQGJ5G424DSFNR3F5TSDO2O7YPW6GJFA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC, address:104.36.82.23:4473): protocol: rpc: dial tcp 104.36.82.23:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.752927386 (190h25m39.184007517s) [noise=true , address=104.36.82.4:4812 , piece_id=GKEQOCQEXDCW3W7NRAGR25GATAKM5TRF55QZFVQSVN4KB5VXGC7Q, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: rpc: dial tcp 104.36.82.4:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.752957093 (190h25m39.184037224s) [noise=true , address=104.36.82.44:4891 , piece_id=CZILE233FG5QMOAHD4H2FDOID4DWUM45QHICCUPORKRACIGNKQJA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753099186 (190h25m39.184179307s) [noise=true , address=104.36.82.5:4841 , piece_id=KFCNOZCXDJMRE5GOOLI5JZNQTDGTFKNUAMXQWVKSNKLFU7CDTCYA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:104.36.82.5:4841): protocol: rpc: dial tcp 104.36.82.5:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753133521 (190h25m39.184213652s) [noise=true , address=104.36.82.58:4293 , piece_id=V42QWQHQPDR45TRH7DKEPM3IYHPGTUQGB3TXSO4TXR4WC6CZKWDQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 188.166.225.170:32786->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753160883 (190h25m39.184241004s) [noise=true , address=104.36.82.60:4339 , piece_id=SLTM2YLQ7S7K47X4D75O6Q4PVCRLNA73CIPLOESPVMYLJANCERCQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: rpc: dial tcp 104.36.82.60:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753179539 (190h25m39.18425967s) [noise=true , address=104.36.82.36:4622 , piece_id=3ZIUVEUCKS7Z2435GLXGAKLXHCOQSGVRFQX7SZRVLL7JUOXU6AYA, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:104.36.82.36:4622): protocol: rpc: dial tcp 104.36.82.36:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753202182 (190h25m39.184282313s) [noise=true , address=104.36.82.59:4313 , piece_id=GOGYV5U2UKUJBJM5DFW3QY277KREG22TF6OWH6TKUKEMQI6OC2VQ, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:129PGPQsUffpwc81NKDdgn5VoYZGmhkBoQpGmDbTpNJsQbfhTvw, address:104.36.82.59:4313): protocol: rpc: dial tcp 104.36.82.59:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753210168 (190h25m39.184290299s) [noise=true , address=104.36.82.35:4605 , piece_id=XPU7NC75Q4JG6N3H35AEKWAMMQIXYKLGDCN6ERELB3G3ES3QQN6A, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:104.36.82.35:4605): protocol: rpc: dial tcp 104.36.82.35:4605: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753232421 (190h25m39.184312552s) [noise=true , address=104.36.82.39:4749 , piece_id=XBGMDHL6EO5ENX2AIPRCEKIWJX3KNSEDFLEZGKTMYRLXJFIYX33A, seg_pos={0 0}, upload=198092]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:104.36.82.39:4749): protocol: write tcp 188.166.225.170:49970->104.36.82.39:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:44.753253440 (190h25m39.184333571s) [seg_pos={0 0}, upload=198092]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:51:44.978448678 (190h25m39.409528809s) [upload=198092]: Done uploading segments... Aug 16 15:51:44.978474928 (190h25m39.409555059s) [upload=198092]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.22718405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 46.654335597s FAIL (2025-08-16 15:51:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.654349873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.289668249s FAIL (2025-08-16 15:51:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:50:24.763679838 (190h24m19.194760009s) [upload=198089]: Starting upload Aug 16 15:50:24.763761825 (190h24m19.194841956s) [upload=198089]: Done starting upload Aug 16 15:50:24.763883106 (190h24m19.194963237s) [upload=198089]: Uploading segments... Aug 16 15:50:24.763988127 (190h24m19.195068248s) [upload=198089]: Got next segment. Inline: false Aug 16 15:50:24.764037973 (190h24m19.195118104s) [upload=198089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:50:24.764044625 (190h24m19.195124746s) [upload=198089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:50:26.030073513 (190h24m20.461153644s) [seg_pos={0 0}, upload=198089]: Begin upload segment... Aug 16 15:50:26.030122697 (190h24m20.461202818s) [seg_pos={0 0}, upload=198089]: Done begin upload segment. Aug 16 15:50:26.030136524 (190h24m20.461216645s) [noise=true , address=104.36.82.9:4025 , piece_id=4LB7SSBLMQIYLVRGBJHRF7ZZSEIXNUUQHMBRCPBBF5ZYXYGEDWIQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.030149488 (190h24m20.461229619s) [noise=true , address=77.90.31.19:4857 , piece_id=JBGN4QS5A6AZRIJXGYCL5QVOVAMDK764NWSSNOBS5EH7XCWY6PWQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.030603417 (190h24m20.461683538s) [noise=true , address=104.36.82.44:4881 , piece_id=U5ACJN6ODQU4EJVSBA6P3QRG2KYY45OESBIZ5ZGDAYMRRRLFM52Q, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.030727694 (190h24m20.461807835s) [noise=true , address=45.91.116.11:4125 , piece_id=SHGCD2UM3O6TJFXILQGK6JE7WA72QDT4KVBVGOMU4RGW4U6BMKBA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.030939440 (190h24m20.462019571s) [noise=true , address=104.36.82.35:4607 , piece_id=3DUDFOZHEWH4SSV3VXWTENCA6LZPJ5BRQZU5TINGKVTIXU2QPSQA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.031218313 (190h24m20.462298444s) [noise=true , address=104.36.82.38:4704 , piece_id=IXQWAYUICRGSIGAYMB3MMOXFTBBNW22TFRAYD5ZQMGAFOLKQ7FRQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.031485034 (190h24m20.462565165s) [noise=true , address=45.91.116.12:4240 , piece_id=4375DRBKKX23C6A2OVQ4TXB2MPC47XSZFTUYE3SMEHE6FXLSIFYA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.031706017 (190h24m20.462786138s) [noise=true , address=77.90.31.16:4764 , piece_id=HGI6EY3GTUISUR5PRFMOUFPDOXR2PL3PQT6U7U4K46S7OFPLFIDA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.031951236 (190h24m20.463031397s) [noise=true , address=77.90.31.12:4607 , piece_id=HJ7EF5NZR7GUSSMGFZJ7NC4VQIDTXUXV7VHRXJQMVCVWCAJ3RKFQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.032167339 (190h24m20.463247471s) [noise=true , address=104.36.82.60:4369 , piece_id=BTQTEMGMTOGVNKJRUKGCMNA7UFS6Z3MOX3MHYA6Z3UIJHO3WYXXQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.032458626 (190h24m20.463538757s) [noise=true , address=79.127.201.215:4344 , piece_id=L2JQ47VXWKRWQB5KUT2AB5JWSZGUPK5EA37EZOW6CIITGQ26TL4A, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.032678948 (190h24m20.463759079s) [noise=true , address=77.90.31.21:4887 , piece_id=OHLCQO7A7OJ52GRJNVBPBIED2MEQS6CBELPHEI2YUQ2ACSRVFIEA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.032916332 (190h24m20.463996463s) [noise=true , address=104.36.82.23:4477 , piece_id=H4PTREQLB4HO4PG7PPDQN2LH3ACFNFA4HPFENMNITS65X3CU4GVQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.033267685 (190h24m20.464347816s) [noise=true , address=66.23.193.175:4557 , piece_id=MPINE4S7QDMTBFJWBPRUFVWUHTGLB7FVADLDI7CRTY52DUA74ATA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.033518084 (190h24m20.464598215s) [noise=true , address=45.91.116.7:4018 , piece_id=VPDN2WWRGVGTHI72TZVDIL4DWT3L7L3BWNM2YNAV4QLSAXHKCOUA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.033843847 (190h24m20.464923978s) [noise=true , address=104.36.82.2:4760 , piece_id=2V325PJFAOJGT3TSTXUS35V7UNZX5U6O5EAVZZ3DYD35HFMUHZ6A, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.033944811 (190h24m20.465024942s) [noise=true , address=104.36.82.59:4302 , piece_id=4WY553VWAP4NTBOPOMAWAVOL62F67OBYP4K4MPPYQIIE4AYOENQA, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:26.034184078 (190h24m20.465264219s) [noise=true , address=104.36.82.5:4842 , piece_id=KYZ56N4L5A254A4ZP2DBM3ZC24EDFVVA67OREYS2MBB6INLD5QGQ, seg_pos={0 0}, upload=198089]: Uploading piece... Aug 16 15:50:27.185718113 (190h24m21.616798244s) [upload=198089]: Next returned nil segment Aug 16 15:50:27.185732571 (190h24m21.616812702s) [upload=198089]: Waiting for error group managing segments... Aug 16 15:50:30.462673559 (190h24m24.89375369s) [noise=true , address=66.23.193.175:4557 , piece_id=MPINE4S7QDMTBFJWBPRUFVWUHTGLB7FVADLDI7CRTY52DUA74ATA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:30.665034263 (190h24m25.096114394s) [noise=true , address=77.90.31.12:4607 , piece_id=HJ7EF5NZR7GUSSMGFZJ7NC4VQIDTXUXV7VHRXJQMVCVWCAJ3RKFQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:31.107372397 (190h24m25.538452518s) [noise=true , address=45.91.116.12:4240 , piece_id=4375DRBKKX23C6A2OVQ4TXB2MPC47XSZFTUYE3SMEHE6FXLSIFYA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:31.135796133 (190h24m25.566876284s) [noise=true , address=77.90.31.19:4857 , piece_id=JBGN4QS5A6AZRIJXGYCL5QVOVAMDK764NWSSNOBS5EH7XCWY6PWQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:32.919740665 (190h24m27.350820796s) [noise=true , address=79.127.201.215:4344 , piece_id=L2JQ47VXWKRWQB5KUT2AB5JWSZGUPK5EA37EZOW6CIITGQ26TL4A, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:33.065582027 (190h24m27.496662158s) [noise=true , address=77.90.31.21:4887 , piece_id=OHLCQO7A7OJ52GRJNVBPBIED2MEQS6CBELPHEI2YUQ2ACSRVFIEA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:34.168548878 (190h24m28.599628999s) [noise=true , address=45.91.116.7:4018 , piece_id=VPDN2WWRGVGTHI72TZVDIL4DWT3L7L3BWNM2YNAV4QLSAXHKCOUA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:34.260152232 (190h24m28.691232363s) [noise=true , address=77.90.31.16:4764 , piece_id=HGI6EY3GTUISUR5PRFMOUFPDOXR2PL3PQT6U7U4K46S7OFPLFIDA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:50:38.625747611 (190h24m33.056827742s) [noise=true , address=45.91.116.11:4125 , piece_id=SHGCD2UM3O6TJFXILQGK6JE7WA72QDT4KVBVGOMU4RGW4U6BMKBA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: <nil> Aug 16 15:51:24.105576772 (190h25m18.536656903s) [noise=true , address=104.36.82.9:4025 , piece_id=4LB7SSBLMQIYLVRGBJHRF7ZZSEIXNUUQHMBRCPBBF5ZYXYGEDWIQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 188.166.225.170:38640->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105635384 (190h25m18.536715505s) [noise=true , address=104.36.82.60:4369 , piece_id=BTQTEMGMTOGVNKJRUKGCMNA7UFS6Z3MOX3MHYA6Z3UIJHO3WYXXQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:12nLksGXGvDwuTWfnmkceuwZNDmrF17iNepHPdYkkvZG2QQUwG7, address:104.36.82.60:4369): protocol: rpc: dial tcp 104.36.82.60:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105667916 (190h25m18.536748037s) [noise=true , address=104.36.82.59:4302 , piece_id=4WY553VWAP4NTBOPOMAWAVOL62F67OBYP4K4MPPYQIIE4AYOENQA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:104.36.82.59:4302): protocol: rpc: dial tcp 104.36.82.59:4302: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105700648 (190h25m18.536780779s) [noise=true , address=104.36.82.23:4477 , piece_id=H4PTREQLB4HO4PG7PPDQN2LH3ACFNFA4HPFENMNITS65X3CU4GVQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf, address:104.36.82.23:4477): protocol: write tcp 188.166.225.170:59068->104.36.82.23:4477: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105732709 (190h25m18.53681284s) [noise=true , address=104.36.82.5:4842 , piece_id=KYZ56N4L5A254A4ZP2DBM3ZC24EDFVVA67OREYS2MBB6INLD5QGQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: write tcp 188.166.225.170:60302->104.36.82.5:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105881585 (190h25m18.536961716s) [noise=true , address=104.36.82.44:4881 , piece_id=U5ACJN6ODQU4EJVSBA6P3QRG2KYY45OESBIZ5ZGDAYMRRRLFM52Q, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:104.36.82.44:4881): protocol: write tcp 188.166.225.170:53884->104.36.82.44:4881: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105912053 (190h25m18.536992184s) [noise=true , address=104.36.82.35:4607 , piece_id=3DUDFOZHEWH4SSV3VXWTENCA6LZPJ5BRQZU5TINGKVTIXU2QPSQA, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1Ex8eUvL1Hqxd5EffckSvBeZDpGrMGhKENyCPvqHybtQf3YQzM, address:104.36.82.35:4607): protocol: rpc: dial tcp 104.36.82.35:4607: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105918926 (190h25m18.536999047s) [noise=true , address=104.36.82.38:4704 , piece_id=IXQWAYUICRGSIGAYMB3MMOXFTBBNW22TFRAYD5ZQMGAFOLKQ7FRQ, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1225CoSQvGuiWjJZWaXhvvq4gxzMymZqUb9fBrfLWw4RekwAEhp, address:104.36.82.38:4704): protocol: rpc: dial tcp 104.36.82.38:4704: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105947791 (190h25m18.537027922s) [noise=true , address=104.36.82.2:4760 , piece_id=2V325PJFAOJGT3TSTXUS35V7UNZX5U6O5EAVZZ3DYD35HFMUHZ6A, seg_pos={0 0}, upload=198089]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: write tcp 188.166.225.170:43568->104.36.82.2:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:51:24.105962480 (190h25m18.537042611s) [seg_pos={0 0}, upload=198089]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:51:24.392852051 (190h25m18.823932182s) [upload=198089]: Done uploading segments... Aug 16 15:51:24.392875395 (190h25m18.823955526s) [upload=198089]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.289698877s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.........................X......X..XXXXXXXXXX.XX.XXXXXX.....] duration: 21.026634625s FAIL (2025-08-16 15:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 21.026650664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..................X.X..X.....X......................X.......] duration: 2.569044828s FAIL (2025-08-16 15:51:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.569063654s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.253751601s FAIL (2025-08-16 15:51:11 UTC) orbiter warning: context deadline exceeded Duration: 6.253779203s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 41.344193605s FAIL (2025-08-16 15:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.344251534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228256734s FAIL (2025-08-16 15:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 39.896483323s) [noise=true , address=79.127.220.101:4076 , piece_id=VLNJK4DSAE3ZGQBCRC7D4H3QQQB5W4MCFESO67FEP2AHQQU5UFVA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.465540816 (190h23m39.896620947s) [noise=true , address=104.36.82.59:4301 , piece_id=MB4MJSEZW524E63Z5HDWPG7FG76ODMV6KIHEDQIOACKPFIFR6AEA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.465737062 (190h23m39.896817193s) [noise=true , address=66.23.193.175:4553 , piece_id=LLKGM5GOI6TPIROLMUY4OHG6ZVNDG2XW6UEISGL4EEKQDTFPNO5Q, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.465824459 (190h23m39.89690458s) [noise=true , address=104.36.82.5:4866 , piece_id=SAEFD6NBDR4BP4HQYKXQNXGN6KJF4AVWX5XYBM2AWEBCPZXQ4BEQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.466058436 (190h23m39.897138567s) [noise=true , address=45.91.116.13:4187 , piece_id=ODA3ODFOTVA2CZ2KQSFTKODST4V7ODE4CRLG75PFMHB6A4ZDZIXQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.466087131 (190h23m39.897167262s) [noise=true , address=104.36.82.2:4760 , piece_id=C3WZRHAKQU7XXUIRTKQHBBT6PSRURCCLD5LSRWRQB4N76THJYYJQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.466350936 (190h23m39.897431057s) [noise=true , address=77.90.31.15:4700 , piece_id=4TXCJB7XP33ICKU5QQXDG65XZUHEK54Y2XGQZ5QYZWZW4F2QPAZA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.466561139 (190h23m39.89764127s) [noise=true , address=104.36.82.39:4758 , piece_id=KGDGDJEGEP7RMHUKS57DHJTOACCMEXK5OA2LE2BAW2D7TJL35LGQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.466830824 (190h23m39.897910955s) [noise=true , address=104.36.82.44:4895 , piece_id=HMO46YQ3KT7AMXOXVVCL5ZVAHZGWSEHTEUPQSHUWVGNX7VT6YWJQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.467366649 (190h23m39.898446781s) [noise=true , address=77.90.31.11:4573 , piece_id=UYL5GP3LAD6LBS6NSLTVO6PKOCVT5Y2ETEUWNLL7KKGVGEBQ2YRQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.467576221 (190h23m39.898656372s) [noise=true , address=77.90.31.14:4689 , piece_id=VJE3AAHKRTGANEXV6OWGNO4JHOM3F2YLV3RWKHJKQ72I4DM5QZNA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.467713824 (190h23m39.898793945s) [noise=true , address=77.90.31.13:4654 , piece_id=2KJBSEZCAOEB3PBOQ74RIGJVPZXZ7YGJM2APCPQOPMDLBYRK46ZA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.468050959 (190h23m39.89913109s) [noise=true , address=104.36.82.4:4816 , piece_id=HJHQFT2CF346ESK33USDVRGTGGGA4Q66RWU4TAPZV6OPA3KEF55A, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.468332187 (190h23m39.899412318s) [noise=true , address=45.91.116.10:4089 , piece_id=HUUE5UCPYREOAZCK4JKF273F3HP6V4C6HUMK7OWSFXDLJ3Q5HQUQ, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:45.468568078 (190h23m39.89964821s) [noise=true , address=104.36.82.23:4463 , piece_id=YYDTMGZ7KZM4DZU3Z2IE7DYV2XL7XH3BP7E4VMM2NFS3Y257VNWA, seg_pos={0 0}, upload=198076]: Uploading piece... Aug 16 15:49:46.646918623 (190h23m41.077998754s) [upload=198076]: Next returned nil segment Aug 16 15:49:46.646932850 (190h23m41.078012971s) [upload=198076]: Waiting for error group managing segments... Aug 16 15:49:47.774544603 (190h23m42.205624734s) [noise=true , address=79.127.220.101:4076 , piece_id=VLNJK4DSAE3ZGQBCRC7D4H3QQQB5W4MCFESO67FEP2AHQQU5UFVA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:48.834868927 (190h23m43.265949058s) [noise=true , address=77.90.31.12:4623 , piece_id=OIKLKZZTRS5OR64D5PSQKMWFXELJKDLXOU3QZZGUPTXNHW6MYT3Q, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:49.291824171 (190h23m43.722904312s) [noise=true , address=77.90.31.15:4700 , piece_id=4TXCJB7XP33ICKU5QQXDG65XZUHEK54Y2XGQZ5QYZWZW4F2QPAZA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:49.994157930 (190h23m44.425238051s) [noise=true , address=77.90.31.13:4654 , piece_id=2KJBSEZCAOEB3PBOQ74RIGJVPZXZ7YGJM2APCPQOPMDLBYRK46ZA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:51.794190445 (190h23m46.225270576s) [noise=true , address=77.90.31.14:4689 , piece_id=VJE3AAHKRTGANEXV6OWGNO4JHOM3F2YLV3RWKHJKQ72I4DM5QZNA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:53.224537929 (190h23m47.65561806s) [noise=true , address=66.23.193.175:4553 , piece_id=LLKGM5GOI6TPIROLMUY4OHG6ZVNDG2XW6UEISGL4EEKQDTFPNO5Q, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:53.290471486 (190h23m47.721551617s) [noise=true , address=45.91.116.10:4089 , piece_id=HUUE5UCPYREOAZCK4JKF273F3HP6V4C6HUMK7OWSFXDLJ3Q5HQUQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:53.509240606 (190h23m47.940320737s) [noise=true , address=77.90.31.11:4573 , piece_id=UYL5GP3LAD6LBS6NSLTVO6PKOCVT5Y2ETEUWNLL7KKGVGEBQ2YRQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:49:55.357161240 (190h23m49.788241371s) [noise=true , address=45.91.116.13:4187 , piece_id=ODA3ODFOTVA2CZ2KQSFTKODST4V7ODE4CRLG75PFMHB6A4ZDZIXQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: <nil> Aug 16 15:50:44.290894099 (190h24m38.721974241s) [noise=true , address=104.36.82.34:4549 , piece_id=ICHJQUGFQPS62BVG4SUUAPZE2RGCRTTTM3NEV7LXJJJRTE55BGCA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:12tn1Sd8Gk3eHSmToBnPGjCyaAdQpUQrhiGe5cuecwBCpd7kEyT, address:104.36.82.34:4549): protocol: write tcp 188.166.225.170:42800->104.36.82.34:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.290931811 (190h24m38.722011932s) [noise=true , address=104.36.82.60:4364 , piece_id=PJNAJ6GEX4WNGVF7JWU7MNLMO5WAHLHTTCPIVTBFIESJYK4TODXA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk, address:104.36.82.60:4364): protocol: write tcp 188.166.225.170:59196->104.36.82.60:4364: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291006024 (190h24m38.722086155s) [noise=true , address=104.36.82.59:4301 , piece_id=MB4MJSEZW524E63Z5HDWPG7FG76ODMV6KIHEDQIOACKPFIFR6AEA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, address:104.36.82.59:4301): protocol: write tcp 188.166.225.170:56976->104.36.82.59:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291048696 (190h24m38.722128827s) [noise=true , address=104.36.82.23:4463 , piece_id=YYDTMGZ7KZM4DZU3Z2IE7DYV2XL7XH3BP7E4VMM2NFS3Y257VNWA, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: write tcp 188.166.225.170:53898->104.36.82.23:4463: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291079705 (190h24m38.722159836s) [noise=true , address=104.36.82.5:4866 , piece_id=SAEFD6NBDR4BP4HQYKXQNXGN6KJF4AVWX5XYBM2AWEBCPZXQ4BEQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:41476->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291118348 (190h24m38.722198469s) [noise=true , address=104.36.82.2:4760 , piece_id=C3WZRHAKQU7XXUIRTKQHBBT6PSRURCCLD5LSRWRQB4N76THJYYJQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: write tcp 188.166.225.170:51970->104.36.82.2:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291150570 (190h24m38.722230701s) [noise=true , address=104.36.82.4:4816 , piece_id=HJHQFT2CF346ESK33USDVRGTGGGA4Q66RWU4TAPZV6OPA3KEF55A, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:12mJkBoAJW49vAg52Ts4EgdmC6BoknUngs6ZqC136rGzdSdVLUy, address:104.36.82.4:4816): protocol: write tcp 188.166.225.170:56260->104.36.82.4:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291194084 (190h24m38.722274215s) [noise=true , address=104.36.82.39:4758 , piece_id=KGDGDJEGEP7RMHUKS57DHJTOACCMEXK5OA2LE2BAW2D7TJL35LGQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:12XjsLCQGSuQkDmyC4QFENshoLeo4kCZG1YNm34wBvG8UEuxTG7, address:104.36.82.39:4758): protocol: write tcp 188.166.225.170:60594->104.36.82.39:4758: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291198582 (190h24m38.722278723s) [noise=true , address=104.36.82.44:4895 , piece_id=HMO46YQ3KT7AMXOXVVCL5ZVAHZGWSEHTEUPQSHUWVGNX7VT6YWJQ, seg_pos={0 0}, upload=198076]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: write tcp 188.166.225.170:44088->104.36.82.44:4895: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:44.291232627 (190h24m38.722312768s) [seg_pos={0 0}, upload=198076]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:50:44.516786419 (190h24m38.94786657s) [upload=198076]: Done uploading segments... Aug 16 15:50:44.516817418 (190h24m38.947897559s) [upload=198076]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.228274368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.15816817s FAIL (2025-08-16 15:50:43 UTC) orbiter warning: context deadline exceeded Duration: 6.1581887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [X..................X.X..X.....X......................X......] duration: 2.148383404s FAIL (2025-08-16 15:50:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.148407639s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.259368963s FAIL (2025-08-16 15:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 15:49:18.125358667 (190h23m12.556438798s) [upload=198071]: Starting upload Aug 16 15:49:18.125399505 (190h23m12.556479636s) [upload=198071]: Done starting upload Aug 16 15:49:18.125561625 (190h23m12.556641756s) [upload=198071]: Uploading segments... Aug 16 15:49:18.125590791 (190h23m12.556670922s) [upload=198071]: Got next segment. Inline: false Aug 16 15:49:18.125610879 (190h23m12.55669101s) [upload=198071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:49:18.125615037 (190h23m12.556695168s) [upload=198071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:49:18.585090052 (190h23m13.016170173s) [seg_pos={0 0}, upload=198071]: Begin upload segment... Aug 16 15:49:18.585124388 (190h23m13.016204519s) [seg_pos={0 0}, upload=198071]: Done begin upload segment. Aug 16 15:49:18.585147001 (190h23m13.016227132s) [noise=true , address=104.36.82.23:4447 , piece_id=H6VA6L3VO4I6XAHW7WLR3NNNO7QMCV6IVBPJOZ5OZUHVRRNVDFZQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.585478786 (190h23m13.016558907s) [noise=true , address=104.36.82.60:4340 , piece_id=RADNVHQNMAY7SQUMTRNOOV2ORXZS56EQVVL6D3SKXXW5MPMWTVDQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.585727592 (190h23m13.016807723s) [noise=true , address=79.127.201.214:4322 , piece_id=XBJHYDTDPQFN5O67XLFMAPT2G5AZMXVT7PHES4DOXPM5SJBWJNRA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.585861498 (190h23m13.016941629s) [noise=true , address=66.23.193.175:4555 , piece_id=JV6SENFFXS6MAERNW5LTZU77ZO3R5ALTZEJEXHVVDVU3PORNNE5A, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.586159048 (190h23m13.017239169s) [noise=true , address=79.127.201.215:4348 , piece_id=EZWTWQUOU2IZYEUC4JXAI42N2KIPF3QPKZGB45H2MHSVDUWE5KLA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.586220525 (190h23m13.017300686s) [noise=true , address=77.90.31.15:4709 , piece_id=NQWH3EGNWQPXYWG6W3RRKOY6I4UM4OQHSKKVUF7GZ77NYQW46JSQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.586501803 (190h23m13.017581934s) [noise=true , address=77.90.31.12:4612 , piece_id=J62ZH7FEZKX5ZSQZFPF4BZX4I2C5SJZL5M5X2FQXL5LXF65SBKRQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.586782060 (190h23m13.017862191s) [noise=true , address=104.36.82.35:4611 , piece_id=IIYHQ2TWP4QBEBGW454TJX5KWY2O3PH5MYC7HP6IAEGKRWXYOCUQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.586837827 (190h23m13.017917968s) [noise=true , address=79.127.220.101:4073 , piece_id=5THPU2SKZELSITNBMOPJA45BFJWQXBRZ4SVNXQENSOODYR5FPHMQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.587039442 (190h23m13.018119563s) [noise=true , address=104.36.82.18:4390 , piece_id=RCGT2Q2W33SE5YBNMHP4EZV2FPJWIYRMCZJ2ICOWODVB7ABIPCJQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.587295622 (190h23m13.018375753s) [noise=true , address=104.36.82.44:4895 , piece_id=D7FIHXFRCL3Y4NUDO27WDV2CYCYKNTKLYODT7E6PHNZP76AP44TA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.587589214 (190h23m13.018669345s) [noise=true , address=77.90.31.19:4866 , piece_id=XAS7KXNIFD33SQF3AGSDQU62ACSIT32OWL6WHU77RJ6Z5LAYW67A, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.587885511 (190h23m13.018965632s) [noise=true , address=45.91.116.11:4125 , piece_id=LWCCEM3DULX5NQY22ATZ5STFEG2R3NEQYYUUKJYGOTOG6TJJ4OBQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.588083460 (190h23m13.019163591s) [noise=true , address=104.36.82.5:4870 , piece_id=CM5G73U4QTRHF4THLLETV2B7QC5CPCAD6CECLHMDLGFHU5UVJ2DQ, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.588303651 (190h23m13.019384083s) [noise=true , address=77.90.31.16:4745 , piece_id=BV72K3NKNHO4JAM26RPWYYVIJVR6I3RMNJ2PNSXX365AQUZXIA3A, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.588509625 (190h23m13.019589757s) [noise=true , address=77.90.31.13:4648 , piece_id=I4ECDHYYQ3GT5IC6H3K2DQGCFJIEWJAN2YXPQYP72KJNMBLS3KMA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.588697886 (190h23m13.019778007s) [noise=true , address=104.36.82.34:4555 , piece_id=U2BA256P2OEYYQPJJOSH5WGZ2QF76MZMQGPB4SJQF22Q2DY2S4YA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:18.588887708 (190h23m13.019967829s) [noise=true , address=104.36.82.38:4698 , piece_id=YM5BWGSNX6YLRXWEK27CZ2ZFBXHQJ5FKUDNBCIFNAYCSCVUYEPXA, seg_pos={0 0}, upload=198071]: Uploading piece... Aug 16 15:49:19.641555904 (190h23m14.072636035s) [noise=true , address=79.127.201.214:4322 , piece_id=XBJHYDTDPQFN5O67XLFMAPT2G5AZMXVT7PHES4DOXPM5SJBWJNRA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:1as6aZfjv58bggvmgDjz5WYupUoWsGaXG1EueYy2TuaM8A8X2e, address:79.127.201.214:4322): protocol: storage node overloaded, request limit: 2000 Aug 16 15:49:19.745406891 (190h23m14.176487022s) [upload=198071]: Next returned nil segment Aug 16 15:49:19.745425357 (190h23m14.176505478s) [upload=198071]: Waiting for error group managing segments... Aug 16 15:49:21.824535085 (190h23m16.255615207s) [noise=true , address=77.90.31.15:4709 , piece_id=NQWH3EGNWQPXYWG6W3RRKOY6I4UM4OQHSKKVUF7GZ77NYQW46JSQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:21.884564865 (190h23m16.315644996s) [noise=true , address=77.90.31.12:4612 , piece_id=J62ZH7FEZKX5ZSQZFPF4BZX4I2C5SJZL5M5X2FQXL5LXF65SBKRQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:22.035322203 (190h23m16.466402334s) [noise=true , address=79.127.220.101:4073 , piece_id=5THPU2SKZELSITNBMOPJA45BFJWQXBRZ4SVNXQENSOODYR5FPHMQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:22.433427394 (190h23m16.864507525s) [noise=true , address=77.90.31.19:4866 , piece_id=XAS7KXNIFD33SQF3AGSDQU62ACSIT32OWL6WHU77RJ6Z5LAYW67A, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:23.943412647 (190h23m18.374492778s) [noise=true , address=77.90.31.13:4648 , piece_id=I4ECDHYYQ3GT5IC6H3K2DQGCFJIEWJAN2YXPQYP72KJNMBLS3KMA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:25.920226383 (190h23m20.351306514s) [noise=true , address=77.90.31.16:4745 , piece_id=BV72K3NKNHO4JAM26RPWYYVIJVR6I3RMNJ2PNSXX365AQUZXIA3A, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:26.729348249 (190h23m21.16042838s) [noise=true , address=66.23.193.175:4555 , piece_id=JV6SENFFXS6MAERNW5LTZU77ZO3R5ALTZEJEXHVVDVU3PORNNE5A, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:29.595270307 (190h23m24.026350439s) [noise=true , address=45.91.116.11:4125 , piece_id=LWCCEM3DULX5NQY22ATZ5STFEG2R3NEQYYUUKJYGOTOG6TJJ4OBQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:49:31.052474328 (190h23m25.483554459s) [noise=true , address=79.127.201.215:4348 , piece_id=EZWTWQUOU2IZYEUC4JXAI42N2KIPF3QPKZGB45H2MHSVDUWE5KLA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: <nil> Aug 16 15:50:17.458438220 (190h24m11.889518351s) [noise=true , address=104.36.82.18:4390 , piece_id=RCGT2Q2W33SE5YBNMHP4EZV2FPJWIYRMCZJ2ICOWODVB7ABIPCJQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: rpc: dial tcp 104.36.82.18:4390: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458481733 (190h24m11.889561864s) [noise=true , address=104.36.82.44:4895 , piece_id=D7FIHXFRCL3Y4NUDO27WDV2CYCYKNTKLYODT7E6PHNZP76AP44TA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: write tcp 188.166.225.170:40164->104.36.82.44:4895: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458531619 (190h24m11.88961175s) [noise=true , address=104.36.82.23:4447 , piece_id=H6VA6L3VO4I6XAHW7WLR3NNNO7QMCV6IVBPJOZ5OZUHVRRNVDFZQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 188.166.225.170:60852->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458569361 (190h24m11.889649492s) [noise=true , address=104.36.82.60:4340 , piece_id=RADNVHQNMAY7SQUMTRNOOV2ORXZS56EQVVL6D3SKXXW5MPMWTVDQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC, address:104.36.82.60:4340): protocol: rpc: dial tcp 104.36.82.60:4340: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458614347 (190h24m11.889694478s) [noise=true , address=104.36.82.34:4555 , piece_id=U2BA256P2OEYYQPJJOSH5WGZ2QF76MZMQGPB4SJQF22Q2DY2S4YA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:129JPGkrFLo95oqnbyr55FBPqPts6UPzX8VQpHf167o5FQV8WFD, address:104.36.82.34:4555): protocol: rpc: dial tcp 104.36.82.34:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458662209 (190h24m11.88974234s) [noise=true , address=104.36.82.35:4611 , piece_id=IIYHQ2TWP4QBEBGW454TJX5KWY2O3PH5MYC7HP6IAEGKRWXYOCUQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc, address:104.36.82.35:4611): protocol: rpc: dial tcp 104.36.82.35:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458695502 (190h24m11.889775633s) [noise=true , address=104.36.82.5:4870 , piece_id=CM5G73U4QTRHF4THLLETV2B7QC5CPCAD6CECLHMDLGFHU5UVJ2DQ, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:1MmdhnDp6kKktFb5btZUDeVQ2u3CNUKCZU7RfuEqiSQ2TEqwKX, address:104.36.82.5:4870): protocol: rpc: dial tcp 104.36.82.5:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458726371 (190h24m11.889806502s) [noise=true , address=104.36.82.38:4698 , piece_id=YM5BWGSNX6YLRXWEK27CZ2ZFBXHQJ5FKUDNBCIFNAYCSCVUYEPXA, seg_pos={0 0}, upload=198071]: Done uploading piece. err: ecclient: upload failed (node:12hbvyP68MKUCvYEhi1kFD33zSRf4oxJgH6weHRnYiYKX9QEC1Q, address:104.36.82.38:4698): protocol: rpc: dial tcp 104.36.82.38:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:50:17.458760848 (190h24m11.889840979s) [seg_pos={0 0}, upload=198071]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:50:17.716113631 (190h24m12.147193762s) [upload=198071]: Done uploading segments... Aug 16 15:50:17.716136826 (190h24m12.147216947s) [upload=198071]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.259381357s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.X..................X.X..X.....X......................X.....] duration: 2.330400416s FAIL (2025-08-16 15:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.330429733s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..........................X......X..XXXXXXXXXX.XX.XXXXXX....] duration: 9.267572477s FAIL (2025-08-16 15:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 9.26760587s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.159415809s FAIL (2025-08-16 15:49:56 UTC) orbiter warning: context deadline exceeded Duration: 6.159430205s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 28.997322277s FAIL (2025-08-16 15:49:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.9973442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...........................X......X..XXXXXXXXXX.XX.XXXXXX...] duration: 17.53801683s FAIL (2025-08-16 15:49:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 17.538046767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232392711s FAIL (2025-08-16 15:49:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 90.31.21:4889 , piece_id=OBQR5EFO6KSJXY3QZ25MGUESCHWRLHSBBRBTKGNXLSVITUCAIPIQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942003561 (190h22m39.373083692s) [noise=true , address=77.90.31.11:4581 , piece_id=BTS23KS75LU4HJBSXP3KHKZVUWEGJAT2IWR6PDWRXH44KVVRFX2Q, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942032798 (190h22m39.373112929s) [noise=true , address=104.36.82.60:4352 , piece_id=G3AEE3XM4YRQAWB6RXL3Q2CBEHMKCFALAFP56H6APN62VE3G5FYQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942335978 (190h22m39.373416109s) [noise=true , address=45.91.116.5:4315 , piece_id=P7SXPQZ2ACIFKRBEUDKIBYE6SGJFGTRF25UOWCH5UJL7WYI4H67Q, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942346167 (190h22m39.373426288s) [noise=true , address=104.36.82.59:4321 , piece_id=USJNY7DBB6T3BK2LYOKVSME5O6YKZKY2ACATUB3BTCPY5BAKOY6A, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942610163 (190h22m39.373690294s) [noise=true , address=79.127.201.215:4354 , piece_id=ZUCAIYXKDTY2RDVDOS4ERCKDT5T2ATRKWOXGCZYZXX6HW6B3KYTQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942618579 (190h22m39.37369871s) [noise=true , address=104.36.82.36:4625 , piece_id=G4YSP6J2XAESACWHCXGVNP4T7SJKHGPYQIHWO3OHFBQXVAE6KVVA, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942866633 (190h22m39.373946764s) [noise=true , address=104.36.82.2:4752 , piece_id=RMTZGWVEICZCC62GNP4P7KP5SZHZGIY4UROAJFXMAEHBXFHCZ5BQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.942912500 (190h22m39.373992621s) [noise=true , address=104.36.82.19:4430 , piece_id=5VLBQLK4QTUACSXEZJRZ6SUEGJCLEO3JVIOGCBBQWYOHONDZYS5Q, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.943104097 (190h22m39.374184228s) [noise=true , address=45.91.116.7:4018 , piece_id=5HM476YSPOIEAQCBFTPXQ6WS2D7NKUUC5U4UUSS4GBK6SNNU5OHQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.943195552 (190h22m39.374275684s) [noise=true , address=104.36.82.5:4871 , piece_id=IOT74K5PVFBELNFREUIZA6DPN4PAE7HA2VDLVORRLKODEFVPBK7A, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.943364896 (190h22m39.374445037s) [noise=true , address=104.36.82.44:4878 , piece_id=APCUZVCJHBVN6LSVFIWMHJB2L4I2WMDDKMGMXTGFUTP33DHZB6AQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.943441583 (190h22m39.374521714s) [noise=true , address=66.23.193.175:4564 , piece_id=ADXEEFZZJS6Y7ZVLRDOYJAWSXNCCXKUTZ6XE7O36IEHULIT4EZ2A, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.943706960 (190h22m39.374787091s) [noise=true , address=104.36.82.35:4595 , piece_id=IC3NKNPHHUQM737PQI4W2NOLRCSEKW26YYBVY57EJEZNMXTNAPJA, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.944084102 (190h22m39.375164233s) [noise=true , address=77.90.31.15:4716 , piece_id=FTAVJ5BBKJ4FQ3T5N4WUK2NP6G7HT53PTZO2HHLKRX4BYGRXJQWQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.944370661 (190h22m39.375450792s) [noise=true , address=104.36.82.23:4453 , piece_id=SM3WBVZHNDSVFMJPY5GYXQRNHN52NJXR6KDS7NM6JH4Y3F3ACO2Q, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:44.944920894 (190h22m39.376001025s) [noise=true , address=77.90.31.12:4604 , piece_id=5TLH5RP43VPHG5TINVFEZ3QKIOXIDYZ5AS5AZ5M4WCGTHCUNXNJQ, seg_pos={0 0}, upload=198061]: Uploading piece... Aug 16 15:48:46.148421410 (190h22m40.579501541s) [upload=198061]: Next returned nil segment Aug 16 15:48:46.148437881 (190h22m40.579518012s) [upload=198061]: Waiting for error group managing segments... Aug 16 15:48:48.228413307 (190h22m42.659493428s) [noise=true , address=79.127.220.101:4076 , piece_id=XL4E3WGCF76BYNBDOTZYLM5LDRJQGIUQRVQEOSQWEREUIWFO4CVA, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:48.500412438 (190h22m42.931492569s) [noise=true , address=79.127.201.215:4354 , piece_id=ZUCAIYXKDTY2RDVDOS4ERCKDT5T2ATRKWOXGCZYZXX6HW6B3KYTQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:49.378903155 (190h22m43.809983286s) [noise=true , address=45.91.116.5:4315 , piece_id=P7SXPQZ2ACIFKRBEUDKIBYE6SGJFGTRF25UOWCH5UJL7WYI4H67Q, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:52.641015169 (190h22m47.0720953s) [noise=true , address=45.91.116.7:4018 , piece_id=5HM476YSPOIEAQCBFTPXQ6WS2D7NKUUC5U4UUSS4GBK6SNNU5OHQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:52.948792337 (190h22m47.379872468s) [noise=true , address=66.23.193.175:4564 , piece_id=ADXEEFZZJS6Y7ZVLRDOYJAWSXNCCXKUTZ6XE7O36IEHULIT4EZ2A, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:55.385035872 (190h22m49.816116053s) [noise=true , address=77.90.31.21:4889 , piece_id=OBQR5EFO6KSJXY3QZ25MGUESCHWRLHSBBRBTKGNXLSVITUCAIPIQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:48:59.285946125 (190h22m53.717026256s) [noise=true , address=77.90.31.12:4604 , piece_id=5TLH5RP43VPHG5TINVFEZ3QKIOXIDYZ5AS5AZ5M4WCGTHCUNXNJQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:49:20.295168681 (190h23m14.726248812s) [noise=true , address=77.90.31.15:4716 , piece_id=FTAVJ5BBKJ4FQ3T5N4WUK2NP6G7HT53PTZO2HHLKRX4BYGRXJQWQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:49:34.861682772 (190h23m29.292762903s) [noise=true , address=77.90.31.11:4581 , piece_id=BTS23KS75LU4HJBSXP3KHKZVUWEGJAT2IWR6PDWRXH44KVVRFX2Q, seg_pos={0 0}, upload=198061]: Done uploading piece. err: <nil> Aug 16 15:49:43.835704313 (190h23m38.266784444s) [noise=true , address=104.36.82.35:4595 , piece_id=IC3NKNPHHUQM737PQI4W2NOLRCSEKW26YYBVY57EJEZNMXTNAPJA, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:12amZR7XxhSNjYTK414HmFCTkMz6DDhT8AjvS1Xcg5C9nzvEK8d, address:104.36.82.35:4595): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:49:43.835782181 (190h23m38.266862312s) [noise=true , address=104.36.82.36:4625 , piece_id=G4YSP6J2XAESACWHCXGVNP4T7SJKHGPYQIHWO3OHFBQXVAE6KVVA, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1G6VA95626cVYC8TDGiqRQAfXYhywwcSy6YieMQoB6tqEvjSrF, address:104.36.82.36:4625): protocol: rpc: dial tcp 104.36.82.36:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.835832958 (190h23m38.266913089s) [noise=true , address=104.36.82.2:4752 , piece_id=RMTZGWVEICZCC62GNP4P7KP5SZHZGIY4UROAJFXMAEHBXFHCZ5BQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:12oRYWQwoe5JtDCqfSShqtyRWVQLjL4UtbokBgNpDAocBKv1vqL, address:104.36.82.2:4752): protocol: rpc: dial tcp 104.36.82.2:4752: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.835932359 (190h23m38.26701249s) [noise=true , address=104.36.82.59:4321 , piece_id=USJNY7DBB6T3BK2LYOKVSME5O6YKZKY2ACATUB3BTCPY5BAKOY6A, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1bN25eEEjxCWKJSr8qj7moDJxArgfsjzPTdBcLdfvoeyxiqPpc, address:104.36.82.59:4321): protocol: rpc: dial tcp 104.36.82.59:4321: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.835998204 (190h23m38.267078335s) [noise=true , address=104.36.82.19:4430 , piece_id=5VLBQLK4QTUACSXEZJRZ6SUEGJCLEO3JVIOGCBBQWYOHONDZYS5Q, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:12eFSmuQiw4ri76UStE1xqEKUtavfyeumrf9MxSNimJ16rGddqo, address:104.36.82.19:4430): protocol: rpc: dial tcp 104.36.82.19:4430: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.836050414 (190h23m38.267130545s) [noise=true , address=104.36.82.5:4871 , piece_id=IOT74K5PVFBELNFREUIZA6DPN4PAE7HA2VDLVORRLKODEFVPBK7A, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1NEVGFZtdgMqREnYBPuBbYqwGRxExenRwncwCBLDFhuwdFyWdb, address:104.36.82.5:4871): protocol: rpc: dial tcp 104.36.82.5:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.836109527 (190h23m38.267189658s) [noise=true , address=104.36.82.60:4352 , piece_id=G3AEE3XM4YRQAWB6RXL3Q2CBEHMKCFALAFP56H6APN62VE3G5FYQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi, address:104.36.82.60:4352): protocol: rpc: dial tcp 104.36.82.60:4352: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.836127342 (190h23m38.267207473s) [noise=true , address=104.36.82.23:4453 , piece_id=SM3WBVZHNDSVFMJPY5GYXQRNHN52NJXR6KDS7NM6JH4Y3F3ACO2Q, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ, address:104.36.82.23:4453): protocol: rpc: dial tcp 104.36.82.23:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.836161577 (190h23m38.267241708s) [noise=true , address=104.36.82.44:4878 , piece_id=APCUZVCJHBVN6LSVFIWMHJB2L4I2WMDDKMGMXTGFUTP33DHZB6AQ, seg_pos={0 0}, upload=198061]: Done uploading piece. err: ecclient: upload failed (node:1wX9xRmyv3KWVyJhFmgBt9uYqDU6U768dqHagKZeKBSEnFup5y, address:104.36.82.44:4878): protocol: write tcp 188.166.225.170:55616->104.36.82.44:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:43.836181145 (190h23m38.267261276s) [seg_pos={0 0}, upload=198061]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:49:44.065675978 (190h23m38.496756109s) [upload=198061]: Done uploading segments... Aug 16 15:49:44.065705354 (190h23m38.496785485s) [upload=198061]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.232404503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [..X..................X.X..X.....X......................X....] duration: 1.968266914s FAIL (2025-08-16 15:49:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.968291521s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXXX] duration: 1m0.229748762s FAIL (2025-08-16 15:49:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:48:17.635718242 (190h22m12.066798373s) [upload=198055]: Starting upload Aug 16 15:48:17.635766174 (190h22m12.066846305s) [upload=198055]: Done starting upload Aug 16 15:48:17.635825647 (190h22m12.066905838s) [upload=198055]: Uploading segments... Aug 16 15:48:17.635994160 (190h22m12.067074341s) [upload=198055]: Got next segment. Inline: false Aug 16 15:48:17.636044366 (190h22m12.067124547s) [upload=198055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:48:17.636052671 (190h22m12.067132832s) [upload=198055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:48:18.091522434 (190h22m12.522602565s) [seg_pos={0 0}, upload=198055]: Begin upload segment... Aug 16 15:48:18.091561558 (190h22m12.522641689s) [seg_pos={0 0}, upload=198055]: Done begin upload segment. Aug 16 15:48:18.091580986 (190h22m12.522661117s) [noise=true , address=77.90.31.12:4624 , piece_id=2D5KVTZVLBDLVY65SXCLGQJK5FUJIED4TTZMT5KVNXBDZ2KO4TNA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.091591425 (190h22m12.522671556s) [noise=true , address=104.36.82.59:4320 , piece_id=VR5ZEW54DWCMNF2YJ2JD7G6B2GOBZTLBH6ZCHY2WKRBIKQJ2BNWA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.091657162 (190h22m12.522737293s) [noise=true , address=104.36.82.3:4783 , piece_id=BLFR7LFFQNZ5U46Y4W6TLJF24UIUVYNHW4J4MMH4L7OVSCEGM3KQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.091949039 (190h22m12.523029161s) [noise=true , address=79.127.201.215:4346 , piece_id=QMTKKKZAWZVKRTBGFSZJJRL3LAEA7SHLDI7FQ5PZDT3G6JZZ3X7Q, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092016850 (190h22m12.523096981s) [noise=true , address=104.36.82.58:4272 , piece_id=N5WIC5NSW2IP4IGT5HTOXQA7NJ3IZFT23KPBT7T7QQ63NPSV6CKQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092205982 (190h22m12.523286113s) [noise=true , address=104.36.82.23:4448 , piece_id=W337O64QQISH56OCZP25TKI74RLJJ3FHNQHK5AYRSJWFUZZ2Z4LA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092373472 (190h22m12.523453593s) [noise=true , address=104.36.82.19:4417 , piece_id=FAQUIBZ7UNHD23SG3EECUNBD6G3S4PSQ4POP6JYFAKZVIIWG3ZUQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092493431 (190h22m12.523573562s) [noise=true , address=77.90.31.11:4563 , piece_id=XDLOM23OJUFXQTM5V66XPAE35Q444VE7KBA5CFB2A5JIHZ2WFT4A, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092634231 (190h22m12.523714362s) [noise=true , address=66.23.193.175:4560 , piece_id=BXYLIOHWEH7WCR4UV5QJKTRAWFCZ466TB5ZZTYTXMV4JTXSDUOLQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.092787795 (190h22m12.523867956s) [noise=true , address=79.127.220.101:4071 , piece_id=KTTD5O4O66FSTYNW5577N6BHZPAJKKGADCAQY5XKHHWKVMAX4T7A, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.093009168 (190h22m12.524089299s) [noise=true , address=45.91.116.10:4110 , piece_id=63M2J4XQEYJ6STSY5IYRSJW7GLKAFSAJGSQPXNNVIY62CEPYGOHA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.093113558 (190h22m12.524193689s) [noise=true , address=104.36.82.44:4897 , piece_id=3R73THOQSICX4ZZUSKGZTUHYOA2GGW3EHYBULLGSDD4CWFYM3H3A, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.093324952 (190h22m12.524405073s) [noise=true , address=104.36.82.36:4651 , piece_id=3XXYRVDB5BUZH2FWOHEAZEANCCZOBZHVO4E6BQNUF55ST4CN36AA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.093597473 (190h22m12.524677605s) [noise=true , address=79.127.220.101:4061 , piece_id=V4OII3ZCXTMRJRJDLYCKY66KREPUMKCEYS4VC4UE6DCUHLFK2RLQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.093827975 (190h22m12.524908096s) [noise=true , address=104.36.82.11:4140 , piece_id=3UVRON6IPKO7QRPNRYVTV5HXGZ5LTKHWCPASDO7GTKCH4YY3V3QA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.094042245 (190h22m12.525122376s) [noise=true , address=104.36.82.17:4359 , piece_id=5SYWABZ3RW6XI6XAYTI3VPNJLY4SW5CD6LWBCUO22PB6T2JHWVEA, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.094235846 (190h22m12.525315977s) [noise=true , address=104.36.82.60:4338 , piece_id=3GGHHW523C7BIO4GGTAZRON5RMBWBGRHKOTH3RQHSXERPTHU4TJQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:18.094478489 (190h22m12.52555862s) [noise=true , address=45.91.116.6:4999 , piece_id=CTRCFGHBVRJWVUB554GT2QLTXNL7VWWRCCCXC337VAMMNE5CGEXQ, seg_pos={0 0}, upload=198055]: Uploading piece... Aug 16 15:48:19.192091111 (190h22m13.623171232s) [upload=198055]: Next returned nil segment Aug 16 15:48:19.192108194 (190h22m13.623188315s) [upload=198055]: Waiting for error group managing segments... Aug 16 15:48:21.836406471 (190h22m16.267486632s) [noise=true , address=79.127.220.101:4061 , piece_id=V4OII3ZCXTMRJRJDLYCKY66KREPUMKCEYS4VC4UE6DCUHLFK2RLQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:23.404532328 (190h22m17.835612449s) [noise=true , address=79.127.201.215:4346 , piece_id=QMTKKKZAWZVKRTBGFSZJJRL3LAEA7SHLDI7FQ5PZDT3G6JZZ3X7Q, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:26.259834167 (190h22m20.690914299s) [noise=true , address=79.127.220.101:4071 , piece_id=KTTD5O4O66FSTYNW5577N6BHZPAJKKGADCAQY5XKHHWKVMAX4T7A, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:28.600084919 (190h22m23.03116505s) [noise=true , address=45.91.116.10:4110 , piece_id=63M2J4XQEYJ6STSY5IYRSJW7GLKAFSAJGSQPXNNVIY62CEPYGOHA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:30.681564474 (190h22m25.112644605s) [noise=true , address=66.23.193.175:4560 , piece_id=BXYLIOHWEH7WCR4UV5QJKTRAWFCZ466TB5ZZTYTXMV4JTXSDUOLQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:31.391420978 (190h22m25.822501109s) [noise=true , address=77.90.31.12:4624 , piece_id=2D5KVTZVLBDLVY65SXCLGQJK5FUJIED4TTZMT5KVNXBDZ2KO4TNA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:48:32.742832343 (190h22m27.173912474s) [noise=true , address=45.91.116.6:4999 , piece_id=CTRCFGHBVRJWVUB554GT2QLTXNL7VWWRCCCXC337VAMMNE5CGEXQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: <nil> Aug 16 15:49:16.990166972 (190h23m11.421247103s) [noise=true , address=104.36.82.44:4897 , piece_id=3R73THOQSICX4ZZUSKGZTUHYOA2GGW3EHYBULLGSDD4CWFYM3H3A, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990213011 (190h23m11.421293142s) [noise=true , address=104.36.82.59:4320 , piece_id=VR5ZEW54DWCMNF2YJ2JD7G6B2GOBZTLBH6ZCHY2WKRBIKQJ2BNWA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12HrqyHVr44byroEByGHdTQyma8kYkh5HBqLocKCGEMFM5teqfb, address:104.36.82.59:4320): protocol: rpc: dial tcp 104.36.82.59:4320: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990246134 (190h23m11.421326255s) [noise=true , address=104.36.82.17:4359 , piece_id=5SYWABZ3RW6XI6XAYTI3VPNJLY4SW5CD6LWBCUO22PB6T2JHWVEA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:1MdZDBkhh7vwffrHksdiixu99PiTgmmNAzdUo39YGaEtpuAHdR, address:104.36.82.17:4359): protocol: rpc: dial tcp 104.36.82.17:4359: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990280971 (190h23m11.421361102s) [noise=true , address=104.36.82.58:4272 , piece_id=N5WIC5NSW2IP4IGT5HTOXQA7NJ3IZFT23KPBT7T7QQ63NPSV6CKQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:1yY39UpswsTLoWz4353JykK35fg8KB4AJWPatKGFgqTt4gDMcR, address:104.36.82.58:4272): protocol: write tcp 188.166.225.170:36586->104.36.82.58:4272: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990372496 (190h23m11.421452627s) [noise=true , address=104.36.82.11:4140 , piece_id=3UVRON6IPKO7QRPNRYVTV5HXGZ5LTKHWCPASDO7GTKCH4YY3V3QA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: write tcp 188.166.225.170:46070->104.36.82.11:4140: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990421029 (190h23m11.42150116s) [noise=true , address=104.36.82.23:4448 , piece_id=W337O64QQISH56OCZP25TKI74RLJJ3FHNQHK5AYRSJWFUZZ2Z4LA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i, address:104.36.82.23:4448): protocol: rpc: dial tcp 104.36.82.23:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990457559 (190h23m11.42153769s) [noise=true , address=104.36.82.19:4417 , piece_id=FAQUIBZ7UNHD23SG3EECUNBD6G3S4PSQ4POP6JYFAKZVIIWG3ZUQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:1TKc4R7hmma2b59RP5Qs5SEw8ha8FECxfJyTdz7mq6jao1JTCK, address:104.36.82.19:4417): protocol: rpc: dial tcp 104.36.82.19:4417: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990501914 (190h23m11.421582045s) [noise=true , address=104.36.82.36:4651 , piece_id=3XXYRVDB5BUZH2FWOHEAZEANCCZOBZHVO4E6BQNUF55ST4CN36AA, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12tD1Zfi4ZxujF9pVdgW5k2wnejXsa7tf1u25FuucRXeneGP3dz, address:104.36.82.36:4651): protocol: rpc: dial tcp 104.36.82.36:4651: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990538975 (190h23m11.421619106s) [noise=true , address=104.36.82.60:4338 , piece_id=3GGHHW523C7BIO4GGTAZRON5RMBWBGRHKOTH3RQHSXERPTHU4TJQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:104.36.82.60:4338): protocol: rpc: dial tcp 104.36.82.60:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990576686 (190h23m11.421656817s) [noise=true , address=104.36.82.3:4783 , piece_id=BLFR7LFFQNZ5U46Y4W6TLJF24UIUVYNHW4J4MMH4L7OVSCEGM3KQ, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: rpc: dial tcp 104.36.82.3:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:49:16.990589922 (190h23m11.421670053s) [noise=true , address=77.90.31.11:4563 , piece_id=XDLOM23OJUFXQTM5V66XPAE35Q444VE7KBA5CFB2A5JIHZ2WFT4A, seg_pos={0 0}, upload=198055]: Done uploading piece. err: ecclient: upload failed (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb, address:77.90.31.11:4563): protocol: write tcp 188.166.225.170:41620->77.90.31.11:4563: use of closed network connection; write tcp 188.166.225.170:41620->77.90.31.11:4563: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41620->77.90.31.11:4563: use of closed network connection Aug 16 15:49:16.990612586 (190h23m11.421692717s) [seg_pos={0 0}, upload=198055]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:49:17.217736973 (190h23m11.648817104s) [upload=198055]: Done uploading segments... Aug 16 15:49:17.217762972 (190h23m11.648843103s) [upload=198055]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.229765363s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.656549268s FAIL (2025-08-16 15:49:10 UTC) orbiter warning: context deadline exceeded Duration: 6.656570578s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [...X..................X.X..X.....X......................X...] duration: 1.901256777s FAIL (2025-08-16 15:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.901280782s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............................X......X..XXXXXXXXXX.XX.XXXXXX..] duration: 16.174502128s FAIL (2025-08-16 15:49:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 16.174521836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............................X......X..XXXXXXXXXX.XX.XXXXXX.] duration: 23.199031386s FAIL (2025-08-16 15:48:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 23.199057737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.23323309s FAIL (2025-08-16 15:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== e_id=QZJGVPPV3Z6DIAGFI23LNHLUNCCKNHUBLBUHCMY5O7PEGSX6JTQA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.489858367 (190h21m38.920938488s) [noise=true , address=45.91.116.10:4090 , piece_id=3VPI3EGYUIRFVTJLJ4YP6ZXFWHCTTXQ7BAGOXNF3DJATJMSS3WWQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490129715 (190h21m38.921209846s) [noise=true , address=77.90.31.11:4584 , piece_id=GLTL3VR36MFLEXPIKDJXOEDAASL2WVPS4IAQNYZVBUC3IFJOFALA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490141388 (190h21m38.921221519s) [noise=true , address=77.90.31.12:4615 , piece_id=LEBZNU4UOL3OLGCGHDDNR7U56KO6LIOFHR7SKIWDLGF5KZUCTTNQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490393069 (190h21m38.9214732s) [noise=true , address=77.90.31.16:4745 , piece_id=OGYI5XZMXAIGTGOSZFYKAPYZPBLOL455T36PAEPSEZXEHYPIP6CQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490474525 (190h21m38.921554656s) [noise=true , address=79.127.201.215:4349 , piece_id=PTCUJRBSIK52QWTWR7IIIFJQCHSDHS5EBGPWXL6IBMTAB77X6LZQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490679126 (190h21m38.921759247s) [noise=true , address=104.36.82.44:4895 , piece_id=7YSNZTPCKIX3WJDDS3K35TJAJ25UBH4T4AASJLZ2X5RW2TVNB4TA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490788566 (190h21m38.921868697s) [noise=true , address=77.90.31.19:4844 , piece_id=PRHRD74A3ASZA5WCVPJO65QGOWUJTWNK2KQCJCAUFLKTNNICCVKQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.490895621 (190h21m38.921975752s) [noise=true , address=77.90.31.21:4886 , piece_id=PBQIWJVZHM3JJK2FXVY3KOUVRBS6ZNREFTVFIJNREMZYPCPE7FAA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491077539 (190h21m38.92215767s) [noise=true , address=104.36.82.60:4349 , piece_id=HIW7N2AGJGXSZJZJKJQGJWHW7ANYQ7K3YHXMSLZVTRWWNCLNUUEA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491116884 (190h21m38.922197015s) [noise=true , address=104.36.82.5:4859 , piece_id=XZMBVZTNBYONKUBZGVKFEID3VKLLQFY75L2E3NONCAXX6RPRVVLQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491345612 (190h21m38.922425743s) [noise=true , address=77.90.31.18:4810 , piece_id=H7ZZLLULRZQGAKQFST5VAJZCTBW6CK6D27ZZPVAHPGCJFVCQOQZA, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491456585 (190h21m38.922536716s) [noise=true , address=45.91.116.6:4985 , piece_id=UH3CVPVL6PYK34RJ33I5LL3NHSWOLZWZMMFG6WOZGRFKDMZIOF2Q, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491687526 (190h21m38.922767657s) [noise=true , address=156.146.43.225:4457 , piece_id=B2LX2SFRXP5LIQBKSXHN7ADGTEA5S3U47OYCFCAERM2NUNQ4RIIQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.491777678 (190h21m38.922857809s) [noise=true , address=104.36.82.23:4464 , piece_id=HARBQCA3IP55Q2BD4OKVOXMGDGMXQQSDFFMDNXBY723BBWUYYY7Q, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.492030743 (190h21m38.923110864s) [noise=true , address=104.36.82.59:4304 , piece_id=ZEW5HJUS5V3AFTF6N6USCRCM5Y3626LZSD6O7UAAUZVFIXFJE7CQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:44.492226257 (190h21m38.923306388s) [noise=true , address=66.23.193.175:4553 , piece_id=UAVPMIQ2DP2XYEVZNANEP5DLXSAQVZUVBR6BYP5JRAAX4UOC7KRQ, seg_pos={0 0}, upload=198044]: Uploading piece... Aug 16 15:47:45.317821273 (190h21m39.748901394s) [noise=true , address=66.23.193.175:4553 , piece_id=UAVPMIQ2DP2XYEVZNANEP5DLXSAQVZUVBR6BYP5JRAAX4UOC7KRQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:14ZJ9cf9uqJqtXTkP9VL2KU5YdCJfUVSyJkfRwJ6K7SpRB5zME, address:66.23.193.175:4553): protocol: storage node overloaded, request limit: 2000 Aug 16 15:47:46.268747146 (190h21m40.699827277s) [upload=198044]: Next returned nil segment Aug 16 15:47:46.268763487 (190h21m40.699843618s) [upload=198044]: Waiting for error group managing segments... Aug 16 15:47:47.170409569 (190h21m41.6014897s) [noise=true , address=156.146.43.225:4457 , piece_id=B2LX2SFRXP5LIQBKSXHN7ADGTEA5S3U47OYCFCAERM2NUNQ4RIIQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:51.208994990 (190h21m45.640075121s) [noise=true , address=45.91.116.10:4090 , piece_id=3VPI3EGYUIRFVTJLJ4YP6ZXFWHCTTXQ7BAGOXNF3DJATJMSS3WWQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:53.523178857 (190h21m47.954258988s) [noise=true , address=79.127.201.215:4349 , piece_id=PTCUJRBSIK52QWTWR7IIIFJQCHSDHS5EBGPWXL6IBMTAB77X6LZQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:56.260043107 (190h21m50.691123238s) [noise=true , address=77.90.31.11:4584 , piece_id=GLTL3VR36MFLEXPIKDJXOEDAASL2WVPS4IAQNYZVBUC3IFJOFALA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:57.427001516 (190h21m51.858081647s) [noise=true , address=77.90.31.18:4810 , piece_id=H7ZZLLULRZQGAKQFST5VAJZCTBW6CK6D27ZZPVAHPGCJFVCQOQZA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:59.491139749 (190h21m53.92221988s) [noise=true , address=77.90.31.19:4844 , piece_id=PRHRD74A3ASZA5WCVPJO65QGOWUJTWNK2KQCJCAUFLKTNNICCVKQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:59.508014066 (190h21m53.939094197s) [noise=true , address=45.91.116.6:4985 , piece_id=UH3CVPVL6PYK34RJ33I5LL3NHSWOLZWZMMFG6WOZGRFKDMZIOF2Q, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:47:59.862177086 (190h21m54.293257217s) [noise=true , address=77.90.31.21:4886 , piece_id=PBQIWJVZHM3JJK2FXVY3KOUVRBS6ZNREFTVFIJNREMZYPCPE7FAA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:48:02.966337659 (190h21m57.39741779s) [noise=true , address=77.90.31.12:4615 , piece_id=LEBZNU4UOL3OLGCGHDDNR7U56KO6LIOFHR7SKIWDLGF5KZUCTTNQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: <nil> Aug 16 15:48:43.382974400 (190h22m37.814054531s) [noise=true , address=104.36.82.5:4859 , piece_id=XZMBVZTNBYONKUBZGVKFEID3VKLLQFY75L2E3NONCAXX6RPRVVLQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383046628 (190h22m37.814126759s) [noise=true , address=104.36.82.2:4760 , piece_id=63CSZWJEXULFULQTFMX4GPFCPCBKEPISKHJERVATH4UG664NMQ5Q, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:1pgY3KuDopCrLXxvbVeNowHD9qSh2Gz7qcgUR1GpsxFkuoHhzB, address:104.36.82.2:4760): protocol: rpc: dial tcp 104.36.82.2:4760: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383187468 (190h22m37.814267599s) [noise=true , address=104.36.82.59:4304 , piece_id=ZEW5HJUS5V3AFTF6N6USCRCM5Y3626LZSD6O7UAAUZVFIXFJE7CQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG, address:104.36.82.59:4304): protocol: rpc: dial tcp 104.36.82.59:4304: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383212756 (190h22m37.814292887s) [noise=true , address=77.90.31.16:4745 , piece_id=OGYI5XZMXAIGTGOSZFYKAPYZPBLOL455T36PAEPSEZXEHYPIP6CQ, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: read tcp 188.166.225.170:50804->77.90.31.16:4745: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383261109 (190h22m37.81434125s) [noise=true , address=104.36.82.44:4895 , piece_id=7YSNZTPCKIX3WJDDS3K35TJAJ25UBH4T4AASJLZ2X5RW2TVNB4TA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: rpc: dial tcp 104.36.82.44:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383290545 (190h22m37.814370676s) [noise=true , address=104.36.82.23:4464 , piece_id=HARBQCA3IP55Q2BD4OKVOXMGDGMXQQSDFFMDNXBY723BBWUYYY7Q, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:1mLbGHntspYmsTVg8433HYSdAEMrXjGnDhmmvX1JmmgcxtdLgk, address:104.36.82.23:4464): protocol: write tcp 188.166.225.170:37406->104.36.82.23:4464: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383311926 (190h22m37.814392067s) [noise=true , address=104.36.82.60:4349 , piece_id=HIW7N2AGJGXSZJZJKJQGJWHW7ANYQ7K3YHXMSLZVTRWWNCLNUUEA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:104.36.82.60:4349): protocol: rpc: dial tcp 104.36.82.60:4349: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383356381 (190h22m37.814436512s) [noise=true , address=104.36.82.4:4831 , piece_id=QZJGVPPV3Z6DIAGFI23LNHLUNCCKNHUBLBUHCMY5O7PEGSX6JTQA, seg_pos={0 0}, upload=198044]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:43.383383723 (190h22m37.814463854s) [seg_pos={0 0}, upload=198044]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 15:48:43.613684509 (190h22m38.04476464s) [upload=198044]: Done uploading segments... Aug 16 15:48:43.613713514 (190h22m38.044793645s) [upload=198044]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.233256104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 18.339821453s FAIL (2025-08-16 15:48:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.339868604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.306182139s FAIL (2025-08-16 15:48:17 UTC) orbiter warning: context deadline exceeded Duration: 8.306206004s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............................X......X..XXXXXXXXXX.XX.XXXXXX] duration: 30.255322459s FAIL (2025-08-16 15:48:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== starting upload Aug 16 15:47:34.951406623 (190h21m29.382486754s) [upload=198040]: Uploading segments... Aug 16 15:47:34.951628868 (190h21m29.382708989s) [upload=198040]: Got next segment. Inline: false Aug 16 15:47:34.951677200 (190h21m29.382757332s) [upload=198040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:34.951681789 (190h21m29.38276192s) [upload=198040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:35.194261059 (190h21m29.62534119s) [seg_pos={0 0}, upload=198040]: Begin upload segment... Aug 16 15:47:35.194307537 (190h21m29.625387668s) [seg_pos={0 0}, upload=198040]: Done begin upload segment. Aug 16 15:47:35.194327656 (190h21m29.625407787s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194331713 (190h21m29.625411874s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194687755 (190h21m29.625767886s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194694026 (190h21m29.625774157s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.194948263 (190h21m29.626028405s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195221717 (190h21m29.626301848s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195481022 (190h21m29.626561154s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.195742262 (190h21m29.626822393s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196010375 (190h21m29.627090506s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196234744 (190h21m29.627314875s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196612056 (190h21m29.627692187s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.196889858 (190h21m29.627969989s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197157761 (190h21m29.628237892s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197371099 (190h21m29.62845122s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197645384 (190h21m29.628725515s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:35.197984903 (190h21m29.629065034s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Uploading piece... Aug 16 15:47:36.978341121 (190h21m31.409421252s) [upload=198040]: Next returned nil segment Aug 16 15:47:36.978362030 (190h21m31.409442161s) [upload=198040]: Waiting for error group managing segments... Aug 16 15:47:37.263014482 (190h21m31.694094613s) [noise=true , address=104.36.82.7:4911 , piece_id=UGCKVTCE5FPWQZJ5MOJLF34JHQZZGDJ7WB5FEE4ZRYDYFYXGV7AA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.302724594 (190h21m31.733804725s) [noise=true , address=104.36.82.7:4910 , piece_id=7T2DZZ4ZGWKKAF2Q6YEYR32MKRNAM4BRK45OJH5L3U2AO3VZJ3YQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.310621062 (190h21m31.741701193s) [noise=true , address=104.36.82.15:4259 , piece_id=JONBD65CZP4QCPSPKT3GBL6SAG47BQRQWMUR5NIDEE36OTKGLWSA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:37.314836638 (190h21m31.745916769s) [noise=true , address=104.36.82.6:4874 , piece_id=QF32Q4NM2DXZXG6A7AOEKNNSORGMQZZ6CM6LNW3J67LPXI2T7WWA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:38.690630479 (190h21m33.1217106s) [noise=true , address=104.36.82.12:4151 , piece_id=I6WWR46GROB6TZI4MH2KC3Z2HVOHNGKLNCNG3EJPGHBU4SFBPNPA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:40.104375525 (190h21m34.535455657s) [noise=true , address=45.91.116.4:4939 , piece_id=7JRGPMD2VKAZK4EHNGFY7EHKZ6Z7TSONCUUW5JSLEELXENJGBCIA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:44.946746883 (190h21m39.377827014s) [noise=true , address=77.90.31.18:4807 , piece_id=ATX6OLQ3ZB6XRJV4Y4UUIGODBLP2NUKTE57AH44MWME2E7UBHT4Q, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:47:56.349839834 (190h21m50.780919966s) [noise=true , address=45.91.116.10:4083 , piece_id=Y4CYABXC5KRMHURK333L475WAXHKQKSWKBEU2F3ZXJA3BJO76NEA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: <nil> Aug 16 15:48:04.301255810 (190h21m58.732335941s) [noise=true , address=45.91.116.9:4263 , piece_id=WHIKZP7R7RIJWJHO57DUJHT4KZD6UJYAEBUV27UCOWI3Y6EZMH5A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302036573 (190h21m58.733116704s) [noise=true , address=77.90.31.11:4554 , piece_id=PEYVHAUSYYYUQHUQZOGNHRDYYORQMJ5SY67IWWEBL7RY3REAB4XA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:48:04.302247768 (190h21m58.733327899s) [noise=true , address=77.90.31.15:4697 , piece_id=MMQKPDJMUGYG5PJA2QULVPUSKJ2DVN2SPRHQYLDCKFTBVO7IVHHQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302329564 (190h21m58.733409695s) [noise=true , address=104.36.82.3:4765 , piece_id=4A6Y6PKPM2WTB5DP7OYJUDFBLJ3KRW6GVYX7WS5OOAD7GEX6OGVQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302453070 (190h21m58.733533211s) [noise=true , address=77.90.31.16:4733 , piece_id=NEEN3XS6LSXJRXTXF6NQIVKEVBNAZU6ONYKW77W5ASJV2PZLPBFQ, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46958->77.90.31.16:4733: use of closed network connection Aug 16 15:48:04.302503036 (190h21m58.733583167s) [noise=true , address=104.36.82.9:4006 , piece_id=QWR7LXFJUFVI2TMF4DQP6JGBJK45QFDAE5OGC7EFLVW7U2XYVP4A, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302549054 (190h21m58.733629185s) [noise=true , address=77.90.31.10:4517 , piece_id=N5GUI7UJRYIU7ES3CNQU4LBILTDDHB27TXANK7FQH4VOOTVU5WNA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54998->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302634367 (190h21m58.733714498s) [noise=true , address=104.36.82.8:4972 , piece_id=OWW6FMBTFOVSUFGUMBC4POKADZABJPXYWGCEONOKVPHSRUJCGVQA, seg_pos={0 0}, upload=198040]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:33022->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:04.302663092 (190h21m58.733743233s) [seg_pos={0 0}, upload=198040]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Aug 16 15:48:04.555606311 (190h21m58.986686442s) [upload=198040]: Done uploading segments... Aug 16 15:48:04.555628454 (190h21m58.986708575s) [upload=198040]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.255339292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [....X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXXX] duration: 1m0.240272311s FAIL (2025-08-16 15:48:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:47:02.825075763 (190h20m57.256155895s) [upload=198033]: Starting upload Aug 16 15:47:02.825134846 (190h20m57.256214977s) [upload=198033]: Done starting upload Aug 16 15:47:02.825235669 (190h20m57.2563158s) [upload=198033]: Uploading segments... Aug 16 15:47:02.825427005 (190h20m57.256507136s) [upload=198033]: Got next segment. Inline: false Aug 16 15:47:02.825459478 (190h20m57.256539609s) [upload=198033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:47:02.825465970 (190h20m57.256546101s) [upload=198033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:47:03.454388113 (190h20m57.885468244s) [seg_pos={0 0}, upload=198033]: Begin upload segment... Aug 16 15:47:03.454435114 (190h20m57.885515245s) [seg_pos={0 0}, upload=198033]: Done begin upload segment. Aug 16 15:47:03.454460853 (190h20m57.885540984s) [noise=true , address=104.36.82.5:4856 , piece_id=MWQH3U7VND2ZZBLXOZ6G2NZKL3BFUOZWQYYUW7OXEYK5JPU5BTIA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.454832473 (190h20m57.885912594s) [noise=true , address=77.90.31.10:4532 , piece_id=ZIHBN322CE4JKLTSE4KRXEJUP3N7OKGNHL7WPBVIKNX2BG45Q6SQ, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.454895604 (190h20m57.885975755s) [noise=true , address=45.91.116.11:4122 , piece_id=RHVCWVO5KB3ZDMZFIREYCTJXQO7QM57REZ3FPALYO2CINB2PEVJQ, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.455347399 (190h20m57.88642754s) [noise=true , address=45.91.116.12:4233 , piece_id=LMTNDUGSNLIJVY64J3F3RN7EBP63OBVGGNRSTK272QQQ2V4KHA3A, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.455380001 (190h20m57.886460132s) [noise=true , address=104.36.82.58:4300 , piece_id=RPEWNSD6P3J3C66ESQJ4HEZQKWJ3CQEMJXX77ZHJNVFSPAUZPK3A, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.455692099 (190h20m57.88677223s) [noise=true , address=104.36.82.43:4840 , piece_id=IZ5UP63GCWQHZV66M3O3CH74XGTK5AE3GQYSSWJRKSVDY5NBWMVQ, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.455723749 (190h20m57.88680388s) [noise=true , address=104.36.82.4:4808 , piece_id=7WFGBCLIM6Q6X2GEXHSBH4WM6SYWJEGDFPLLNJOWUA72CUCDXUUA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456047870 (190h20m57.887128001s) [noise=true , address=104.36.82.35:4596 , piece_id=KVT2CBLOR67Y3HGBEYTCAGYI437KS2NGWMRN2LGGKVA7RY5AVG5A, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456058249 (190h20m57.88713838s) [noise=true , address=104.36.82.60:4362 , piece_id=BMCQQ3YAAVYLVIIZHAQY2MVTC2EHT2MIJOOKSSAX6GCVB3TN2BGQ, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456387128 (190h20m57.887467259s) [noise=true , address=104.36.82.44:4891 , piece_id=V25U3GD3SQ2E2ZRBVKTAW727Q35N3M7TYWSJT4K4YG34P3U4X52A, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456490156 (190h20m57.887570297s) [noise=true , address=79.127.220.101:4071 , piece_id=FBFYE7YLYAHPUDDI3XQ7FFUWMDJC5E3QUYIROWYD3P64MMR53ZMA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456670360 (190h20m57.887750491s) [noise=true , address=104.36.82.23:4467 , piece_id=5AJ446B2GIJIRSACHCASRIJ75AMMQKISXDZZIL3K2RIFGFLTVQVA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456904689 (190h20m57.88798482s) [noise=true , address=79.127.201.215:4345 , piece_id=W7WFAVUBDFE542HLDRXII3YDSKLML635RB3IP3G3H6SMJUPI6Z2Q, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.456944785 (190h20m57.888024916s) [noise=true , address=66.23.193.175:4553 , piece_id=UQ7THOI4N6BT2ZH2GXH4DDD5GB4O7SC56KCVEGGSQLI6V676SVDA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.457310185 (190h20m57.888390306s) [noise=true , address=104.36.82.59:4333 , piece_id=PHOCR52YNRHG46OMBEDRHNCAG4RE3EAKL5WDSDM2IBRR3FDUW4BA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.457325153 (190h20m57.888405294s) [noise=true , address=77.90.31.11:4582 , piece_id=42HE7SWV6BHJB6OACHXQ5ZTRSAXT7T2MSLHIRXDJWQOZF76NFAKA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.457587305 (190h20m57.888667436s) [noise=true , address=104.36.82.37:4670 , piece_id=Z4KWUJP7UEDWFC3YUAA4CYLMDJVAH4GUE4SDVSYBJWW6ZG6KXV7A, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:03.457741039 (190h20m57.88882117s) [noise=true , address=104.36.82.43:4854 , piece_id=LVOMY2QRMKZ65ZX3RHGCUM4E6HQ4FBYMT75VLYVDSKRSSM5NBFVA, seg_pos={0 0}, upload=198033]: Uploading piece... Aug 16 15:47:04.552029820 (190h20m58.983109951s) [upload=198033]: Next returned nil segment Aug 16 15:47:04.552051081 (190h20m58.983131242s) [upload=198033]: Waiting for error group managing segments... Aug 16 15:47:05.836530801 (190h21m0.267610932s) [noise=true , address=104.36.82.43:4854 , piece_id=LVOMY2QRMKZ65ZX3RHGCUM4E6HQ4FBYMT75VLYVDSKRSSM5NBFVA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:12eitcwvcNX2GyMiv4GdfDD8wmgb6a1d3ggaGVxbpyKAjPjshmV, address:104.36.82.43:4854): protocol: storage node overloaded, request limit: 2000 Aug 16 15:47:07.252452563 (190h21m1.683532694s) [noise=true , address=79.127.220.101:4071 , piece_id=FBFYE7YLYAHPUDDI3XQ7FFUWMDJC5E3QUYIROWYD3P64MMR53ZMA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:09.883807361 (190h21m4.314887522s) [noise=true , address=104.36.82.43:4840 , piece_id=IZ5UP63GCWQHZV66M3O3CH74XGTK5AE3GQYSSWJRKSVDY5NBWMVQ, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:10.072636153 (190h21m4.503716284s) [noise=true , address=79.127.201.215:4345 , piece_id=W7WFAVUBDFE542HLDRXII3YDSKLML635RB3IP3G3H6SMJUPI6Z2Q, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:13.289174698 (190h21m7.720254829s) [noise=true , address=66.23.193.175:4553 , piece_id=UQ7THOI4N6BT2ZH2GXH4DDD5GB4O7SC56KCVEGGSQLI6V676SVDA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:19.275301233 (190h21m13.706381364s) [noise=true , address=77.90.31.11:4582 , piece_id=42HE7SWV6BHJB6OACHXQ5ZTRSAXT7T2MSLHIRXDJWQOZF76NFAKA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:22.494651276 (190h21m16.925731408s) [noise=true , address=45.91.116.12:4233 , piece_id=LMTNDUGSNLIJVY64J3F3RN7EBP63OBVGGNRSTK272QQQ2V4KHA3A, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:47:24.484375605 (190h21m18.915455736s) [noise=true , address=45.91.116.11:4122 , piece_id=RHVCWVO5KB3ZDMZFIREYCTJXQO7QM57REZ3FPALYO2CINB2PEVJQ, seg_pos={0 0}, upload=198033]: Done uploading piece. err: <nil> Aug 16 15:48:02.131254148 (190h21m56.562334309s) [noise=true , address=104.36.82.44:4891 , piece_id=V25U3GD3SQ2E2ZRBVKTAW727Q35N3M7TYWSJT4K4YG34P3U4X52A, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131329903 (190h21m56.562410044s) [noise=true , address=104.36.82.59:4333 , piece_id=PHOCR52YNRHG46OMBEDRHNCAG4RE3EAKL5WDSDM2IBRR3FDUW4BA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:12K7U6NXcB6SFBtQFrG7DXt2skACr3u5DX2QsrnToi8EKywS9TN, address:104.36.82.59:4333): protocol: rpc: dial tcp 104.36.82.59:4333: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131372735 (190h21m56.562452866s) [noise=true , address=104.36.82.35:4596 , piece_id=KVT2CBLOR67Y3HGBEYTCAGYI437KS2NGWMRN2LGGKVA7RY5AVG5A, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:11HDJFxgZxCiibSDmX5sdwZ1jSEd2NpBPBjMQ858nmRiMGfP7q, address:104.36.82.35:4596): protocol: rpc: dial tcp 104.36.82.35:4596: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131406660 (190h21m56.562486791s) [noise=true , address=104.36.82.60:4362 , piece_id=BMCQQ3YAAVYLVIIZHAQY2MVTC2EHT2MIJOOKSSAX6GCVB3TN2BGQ, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:12238rvPqrDXBowLsUiKTykdZFgfyq4fX9YdaeLa52sQJCkRMHM, address:104.36.82.60:4362): protocol: rpc: dial tcp 104.36.82.60:4362: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131439523 (190h21m56.562519654s) [noise=true , address=104.36.82.37:4670 , piece_id=Z4KWUJP7UEDWFC3YUAA4CYLMDJVAH4GUE4SDVSYBJWW6ZG6KXV7A, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1Tn26F2GhEKm5HP7P7SFYjyBsRt31ueACyGzXdRYoPwEsK7jqp, address:104.36.82.37:4670): protocol: write tcp 188.166.225.170:37040->104.36.82.37:4670: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131484999 (190h21m56.56256513s) [noise=true , address=104.36.82.58:4300 , piece_id=RPEWNSD6P3J3C66ESQJ4HEZQKWJ3CQEMJXX77ZHJNVFSPAUZPK3A, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1g2c35xc8XXpVwwjcNWkqTDgqix5ow9otieWT4A4zi5cHPdWqM, address:104.36.82.58:4300): protocol: rpc: dial tcp 104.36.82.58:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131522271 (190h21m56.562602402s) [noise=true , address=104.36.82.4:4808 , piece_id=7WFGBCLIM6Q6X2GEXHSBH4WM6SYWJEGDFPLLNJOWUA72CUCDXUUA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1fbQCaGY9ZssBzyNusnhyY994e8ytvW4sQDQRt7q2tdGrg6ps3, address:104.36.82.4:4808): protocol: rpc: dial tcp 104.36.82.4:4808: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131553561 (190h21m56.562633692s) [noise=true , address=104.36.82.5:4856 , piece_id=MWQH3U7VND2ZZBLXOZ6G2NZKL3BFUOZWQYYUW7OXEYK5JPU5BTIA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: rpc: dial tcp 104.36.82.5:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131595000 (190h21m56.562675161s) [noise=true , address=104.36.82.23:4467 , piece_id=5AJ446B2GIJIRSACHCASRIJ75AMMQKISXDZZIL3K2RIFGFLTVQVA, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1RegmCcGuZED5cPuhVJ9mLRuouCHBEZ3hecf7aseHsMCgo1ajT, address:104.36.82.23:4467): protocol: rpc: dial tcp 104.36.82.23:4467: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131615409 (190h21m56.56269554s) [noise=true , address=77.90.31.10:4532 , piece_id=ZIHBN322CE4JKLTSE4KRXEJUP3N7OKGNHL7WPBVIKNX2BG45Q6SQ, seg_pos={0 0}, upload=198033]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: write tcp 188.166.225.170:34028->77.90.31.10:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:48:02.131638463 (190h21m56.562718604s) [seg_pos={0 0}, upload=198033]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:48:02.369522769 (190h21m56.8006029s) [upload=198033]: Done uploading segments... Aug 16 15:48:02.369550482 (190h21m56.800630613s) [upload=198033]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.240284023s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 35.714714106s FAIL (2025-08-16 15:47:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.714737131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [X...X..................X.X..X.....X......................X..] duration: 1.929311157s FAIL (2025-08-16 15:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.929345413s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.238798161s FAIL (2025-08-16 15:47:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== FSQTTSXHLQAAWON5GIU7XQ, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552015066 (190h20m33.983095187s) [noise=true , address=104.36.82.60:4337 , piece_id=MSG5BS7CTZHINFPC56VVYAGAA2MLDPEPMEPFREMEH3DPQSVZKMCA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552033011 (190h20m33.983113142s) [noise=true , address=104.36.82.9:4025 , piece_id=DX6ISWAKTEUVR6Y53QNDCRDZTIYOFX2FZCPSWRVCKN2X3NNAC4JQ, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552341230 (190h20m33.983421361s) [noise=true , address=104.36.82.44:4889 , piece_id=POCPQEINNQ5GPY6ZGA6YVE47UD5TU2A5DLH2XCG5MEDB47PEJQYA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552383892 (190h20m33.983464023s) [noise=true , address=104.36.82.34:4566 , piece_id=BOG2I27EUUTB5YRVXBNXRQHN6S5CYCMNBW4FWG33O7ME62LFCTSQ, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552619412 (190h20m33.983699543s) [noise=true , address=104.36.82.4:4815 , piece_id=KRJFNZ64AEBJI6C2KMV6NXQCADLFMZTZWRRN54NL5KJ6S2J7555A, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552707722 (190h20m33.983787853s) [noise=true , address=77.90.31.13:4659 , piece_id=DF2CL6C4RGUSIIS7UA5ZOFK5B6XOOOYKWH77Q7EJJKONGB76WQFA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552884991 (190h20m33.983965122s) [noise=true , address=79.127.220.101:4071 , piece_id=5WC2MQAH3FAWBHD4F3KXJQX34AG7EMSMUZR3FFLTUGMK55P6VDCA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.552993478 (190h20m33.984073609s) [noise=true , address=104.36.82.23:4448 , piece_id=SJ2C6RMNG3SHWLANDDYPDXPJENBC5L6JOBWF6D2CLGO33TXFRZCA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.553085544 (190h20m33.984165675s) [noise=true , address=77.90.31.12:4612 , piece_id=VAD3LT5JUGZR4KBNCBYXDTLINWKLQHQQ3K3BSPCHT7Y7QTYGHLQQ, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.553288012 (190h20m33.984368143s) [noise=true , address=104.36.82.59:4313 , piece_id=XMP7QM4B3XQGPFNHMNWRYH5EV5MSVN3VWI5ZORT2EBMAOLRPPX4A, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.553501800 (190h20m33.984581932s) [noise=true , address=104.36.82.58:4291 , piece_id=HRV3VRQPX4AAP6ONXUOHGSLXD53GTL3N5F3GXMO5S7DKY7XRLTFA, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.554206669 (190h20m33.9852868s) [noise=true , address=104.36.82.51:4064 , piece_id=3DL2HLW4CUEBT7TZCWVLDBFGG2UWAGP6SGITWH6N2QUJ4R7UYRGQ, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:39.554839240 (190h20m33.985919371s) [noise=true , address=45.91.116.5:4324 , piece_id=A6LGLD2HQDQSWRYO3XMBCA64UDXQFRLJA2NXEH5O7IPA3V3KSX6A, seg_pos={0 0}, upload=198024]: Uploading piece... Aug 16 15:46:41.080734815 (190h20m35.511814946s) [upload=198024]: Next returned nil segment Aug 16 15:46:41.080752829 (190h20m35.51183296s) [upload=198024]: Waiting for error group managing segments... Aug 16 15:46:42.528968213 (190h20m36.960048394s) [noise=true , address=79.127.220.101:4071 , piece_id=5WC2MQAH3FAWBHD4F3KXJQX34AG7EMSMUZR3FFLTUGMK55P6VDCA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:46:46.125757182 (190h20m40.556837313s) [noise=true , address=45.91.116.5:4324 , piece_id=A6LGLD2HQDQSWRYO3XMBCA64UDXQFRLJA2NXEH5O7IPA3V3KSX6A, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:46:46.895124873 (190h20m41.326205004s) [noise=true , address=77.90.31.21:4904 , piece_id=6V42EW6LDGY5H2GOYS6INFXVRISCFQ6AA2URTQYIP3JPDMNJ67MQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:46:48.252785669 (190h20m42.6838658s) [noise=true , address=77.90.31.12:4612 , piece_id=VAD3LT5JUGZR4KBNCBYXDTLINWKLQHQQ3K3BSPCHT7Y7QTYGHLQQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:46:52.965410261 (190h20m47.396490392s) [noise=true , address=77.90.31.13:4659 , piece_id=DF2CL6C4RGUSIIS7UA5ZOFK5B6XOOOYKWH77Q7EJJKONGB76WQFA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:47:06.025029834 (190h21m0.456109965s) [noise=true , address=77.90.31.18:4826 , piece_id=6IIYBQOA5XYMIMTRKNHABD76WUYVGMOUNS5MJRP5HNI2T4XU4UEQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: <nil> Aug 16 15:47:38.466265041 (190h21m32.897345162s) [noise=true , address=104.36.82.23:4448 , piece_id=SJ2C6RMNG3SHWLANDDYPDXPJENBC5L6JOBWF6D2CLGO33TXFRZCA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i, address:104.36.82.23:4448): protocol: rpc: dial tcp 104.36.82.23:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466308143 (190h21m32.897388274s) [noise=true , address=104.36.82.51:4064 , piece_id=3DL2HLW4CUEBT7TZCWVLDBFGG2UWAGP6SGITWH6N2QUJ4R7UYRGQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy, address:104.36.82.51:4064): protocol: rpc: dial tcp 104.36.82.51:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466337489 (190h21m32.89741762s) [noise=true , address=104.36.82.34:4566 , piece_id=BOG2I27EUUTB5YRVXBNXRQHN6S5CYCMNBW4FWG33O7ME62LFCTSQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1Z9ic1vvfDViKVZbrkLZnryBxKKQ4rJdjBFY6wEYCDtJt9ci77, address:104.36.82.34:4566): protocol: rpc: dial tcp 104.36.82.34:4566: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466369330 (190h21m32.897449461s) [noise=true , address=104.36.82.60:4337 , piece_id=MSG5BS7CTZHINFPC56VVYAGAA2MLDPEPMEPFREMEH3DPQSVZKMCA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, address:104.36.82.60:4337): protocol: rpc: dial tcp 104.36.82.60:4337: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466388186 (190h21m32.897468317s) [noise=true , address=104.36.82.4:4815 , piece_id=KRJFNZ64AEBJI6C2KMV6NXQCADLFMZTZWRRN54NL5KJ6S2J7555A, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1D4gdLHtXGAiLFhppWTVvqXjmiU6r5KJPyW1HG8zzyZ3EC6eYa, address:104.36.82.4:4815): protocol: rpc: dial tcp 104.36.82.4:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466431008 (190h21m32.897511139s) [noise=true , address=104.36.82.36:4638 , piece_id=M35DNJNDPDPLN2G2GBS5DRUFHVVPF2GKBMFOMU2J5RN7GNB2R75A, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1poiPDhUGb5ZGnZnZcRViY45RefT9anD83YFcqZVZspnnJ9fN5, address:104.36.82.36:4638): protocol: rpc: dial tcp 104.36.82.36:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466482868 (190h21m32.897562999s) [noise=true , address=104.36.82.5:4850 , piece_id=I7D4TBPFFABUXJ74M2OEBDRVIUWGWYD5TOJ3FEJB4GQYH63ULZCQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi, address:104.36.82.5:4850): protocol: rpc: dial tcp 104.36.82.5:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466515410 (190h21m32.897595531s) [noise=true , address=104.36.82.4:4810 , piece_id=ZREHGG7YVCJ2ODWA2YJEHOCX27PQWZFSQTTSXHLQAAWON5GIU7XQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:104.36.82.4:4810): protocol: rpc: dial tcp 104.36.82.4:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466542732 (190h21m32.897622863s) [noise=true , address=104.36.82.58:4291 , piece_id=HRV3VRQPX4AAP6ONXUOHGSLXD53GTL3N5F3GXMO5S7DKY7XRLTFA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:12gn31FU1BnbHPLvB7k1RZhtJbDdJCLyBLa5A6VhDNMFN7erSG1, address:104.36.82.58:4291): protocol: rpc: dial tcp 104.36.82.58:4291: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466562200 (190h21m32.897642321s) [noise=true , address=104.36.82.9:4025 , piece_id=DX6ISWAKTEUVR6Y53QNDCRDZTIYOFX2FZCPSWRVCKN2X3NNAC4JQ, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: rpc: dial tcp 104.36.82.9:4025: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466570946 (190h21m32.897651077s) [noise=true , address=104.36.82.44:4889 , piece_id=POCPQEINNQ5GPY6ZGA6YVE47UD5TU2A5DLH2XCG5MEDB47PEJQYA, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:1PCtJRFHPHi1dHJNcRnGZg8HZPQDfo5L3aw6DgVnUzVUz92Kyv, address:104.36.82.44:4889): protocol: rpc: dial tcp 104.36.82.44:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466591425 (190h21m32.897671557s) [noise=true , address=104.36.82.59:4313 , piece_id=XMP7QM4B3XQGPFNHMNWRYH5EV5MSVN3VWI5ZORT2EBMAOLRPPX4A, seg_pos={0 0}, upload=198024]: Done uploading piece. err: ecclient: upload failed (node:129PGPQsUffpwc81NKDdgn5VoYZGmhkBoQpGmDbTpNJsQbfhTvw, address:104.36.82.59:4313): protocol: rpc: dial tcp 104.36.82.59:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:38.466612054 (190h21m32.897692185s) [seg_pos={0 0}, upload=198024]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 15:47:38.703459476 (190h21m33.134539598s) [upload=198024]: Done uploading segments... Aug 16 15:47:38.703481078 (190h21m33.134561209s) [upload=198024]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.238811316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.183088812s FAIL (2025-08-16 15:47:08 UTC) orbiter warning: context deadline exceeded Duration: 8.18310819s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...............................X......X..XXXXXXXXXX.XX.XXXXX] duration: 30.236685459s FAIL (2025-08-16 15:47:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 16 15:46:33.438091946 (190h20m27.869172077s) [upload=198023]: Starting upload Aug 16 15:46:33.438147092 (190h20m27.869227223s) [upload=198023]: Done starting upload Aug 16 15:46:33.438193279 (190h20m27.86927341s) [upload=198023]: Uploading segments... Aug 16 15:46:33.438309152 (190h20m27.869389283s) [upload=198023]: Got next segment. Inline: false Aug 16 15:46:33.438331654 (190h20m27.869411775s) [upload=198023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:33.438335612 (190h20m27.869415744s) [upload=198023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:46:33.887780937 (190h20m28.318861068s) [seg_pos={0 0}, upload=198023]: Begin upload segment... Aug 16 15:46:33.887819130 (190h20m28.318899261s) [seg_pos={0 0}, upload=198023]: Done begin upload segment. Aug 16 15:46:33.887838467 (190h20m28.318918598s) [noise=true , address=104.36.82.10:4077 , piece_id=Z5UHBBJDIRBQLWZJIIMST5ZDEQZIZZXZAUHEQVQU23BREFH2WN5A, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.887875528 (190h20m28.318955659s) [noise=true , address=104.36.82.12:4151 , piece_id=YQFGAHB7T7BQNAOJ4RRH27UAEIN6KZKCPIEE6EZDNWBKQJXEYCJQ, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888188957 (190h20m28.319269078s) [noise=true , address=104.36.82.3:4765 , piece_id=JUKT6TKJNZ37RTL37K6K35HJ6E646C442NOYKZ34U7QHLA7CS4PA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888206822 (190h20m28.319286943s) [noise=true , address=104.36.82.6:4876 , piece_id=TFGYG62RPDGUWFZN7NXFXC6VECS4TGLZJTDPVBOPWAXFSKEEXMBA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888487018 (190h20m28.319567139s) [noise=true , address=104.36.82.16:4293 , piece_id=GT3A4ZWL2K3D5IUVUU2QARQK2GP2AQU4MJXE2KWTIBV53VHL2D5A, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888529069 (190h20m28.3196092s) [noise=true , address=104.36.82.7:4910 , piece_id=GOB24FLVXOSNK3ISYSZAWH6H64CKIX4LQ3VIHBXQHRBTBTBR65FA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888760150 (190h20m28.319840271s) [noise=true , address=77.90.31.18:4807 , piece_id=3DVXC5VZTA4IT4KSYAYJMJ3W45VQR5V7E3ABGXPZPLOMUCDOL3FA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.888799366 (190h20m28.319879497s) [noise=true , address=45.91.116.10:4082 , piece_id=52DQULH3F7RGYPRX5XNS2KIZN6FQFMGSBFHU4HWTY4BERCKTTHFQ, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889044385 (190h20m28.320124516s) [noise=true , address=77.90.31.13:4628 , piece_id=OBR3GM5DWGSOFAFJ4TKEHAZMGH3SCFAZT3F2ZZUUZHB7MRAAHUUA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889118125 (190h20m28.320198256s) [noise=true , address=104.36.82.8:4969 , piece_id=7M24H4PJJU5B6EUZE2F7MTW745C7VMKNGX43AD5TMKQLNNBQDUFA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889256049 (190h20m28.32033618s) [noise=true , address=45.91.116.4:4938 , piece_id=5YQ3RASLKQZEHOSC6SAKT4PU5EFKDDZAUMB6IMS2UXPOOVVCJF6Q, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889350290 (190h20m28.320430421s) [noise=true , address=104.36.82.7:4911 , piece_id=4IQQOCAS7RN4CTBW7A2EKWOGNTLCFN7B3VMYGOMA2LBQD4AOBYAA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889470790 (190h20m28.320550921s) [noise=true , address=45.91.116.7:4011 , piece_id=PTARTD2LXB2PORXLQHNRYY3GWC3IMTWW5QARFWAC4ATHMYIP4QBA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889568898 (190h20m28.320649029s) [noise=true , address=104.36.82.15:4259 , piece_id=5BXDU3TP2PCRYLF2NDXOWIYEFIDUMYK474JUJO52YFTEIAWUERNA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.889787136 (190h20m28.320867267s) [noise=true , address=77.90.31.10:4518 , piece_id=7QFOYXKKIRRPLXWOMZTJLEOTHHOEQ5GRDFSPHLWNHO7CJ6FA5FDA, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:33.890023317 (190h20m28.321103438s) [noise=true , address=45.91.116.14:4189 , piece_id=5O6WXL32AWIAXB6M5MTURPI4VF22IEUITZSUNCXQ6S2RS73GTZAQ, seg_pos={0 0}, upload=198023]: Uploading piece... Aug 16 15:46:35.729104077 (190h20m30.160184198s) [upload=198023]: Next returned nil segment Aug 16 15:46:35.729119797 (190h20m30.160199928s) [upload=198023]: Waiting for error group managing segments... Aug 16 15:46:36.031632662 (190h20m30.462712794s) [noise=true , address=104.36.82.7:4910 , piece_id=GOB24FLVXOSNK3ISYSZAWH6H64CKIX4LQ3VIHBXQHRBTBTBR65FA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:36.041600414 (190h20m30.472680545s) [noise=true , address=104.36.82.15:4259 , piece_id=5BXDU3TP2PCRYLF2NDXOWIYEFIDUMYK474JUJO52YFTEIAWUERNA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:36.117387334 (190h20m30.548467465s) [noise=true , address=104.36.82.7:4911 , piece_id=4IQQOCAS7RN4CTBW7A2EKWOGNTLCFN7B3VMYGOMA2LBQD4AOBYAA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:36.153756643 (190h20m30.584836804s) [noise=true , address=104.36.82.10:4077 , piece_id=Z5UHBBJDIRBQLWZJIIMST5ZDEQZIZZXZAUHEQVQU23BREFH2WN5A, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:36.284170275 (190h20m30.715250396s) [noise=true , address=104.36.82.6:4876 , piece_id=TFGYG62RPDGUWFZN7NXFXC6VECS4TGLZJTDPVBOPWAXFSKEEXMBA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:43.971150068 (190h20m38.402230189s) [noise=true , address=77.90.31.13:4628 , piece_id=OBR3GM5DWGSOFAFJ4TKEHAZMGH3SCFAZT3F2ZZUUZHB7MRAAHUUA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:44.968483337 (190h20m39.399563478s) [noise=true , address=104.36.82.12:4151 , piece_id=YQFGAHB7T7BQNAOJ4RRH27UAEIN6KZKCPIEE6EZDNWBKQJXEYCJQ, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:48.926731272 (190h20m43.357811403s) [noise=true , address=45.91.116.10:4082 , piece_id=52DQULH3F7RGYPRX5XNS2KIZN6FQFMGSBFHU4HWTY4BERCKTTHFQ, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:49.857459067 (190h20m44.288539188s) [noise=true , address=77.90.31.18:4807 , piece_id=3DVXC5VZTA4IT4KSYAYJMJ3W45VQR5V7E3ABGXPZPLOMUCDOL3FA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:46:58.685840684 (190h20m53.116920815s) [noise=true , address=45.91.116.4:4938 , piece_id=5YQ3RASLKQZEHOSC6SAKT4PU5EFKDDZAUMB6IMS2UXPOOVVCJF6Q, seg_pos={0 0}, upload=198023]: Done uploading piece. err: <nil> Aug 16 15:47:02.786729291 (190h20m57.217809422s) [noise=true , address=45.91.116.7:4011 , piece_id=PTARTD2LXB2PORXLQHNRYY3GWC3IMTWW5QARFWAC4ATHMYIP4QBA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:47:02.786845344 (190h20m57.217925475s) [noise=true , address=104.36.82.3:4765 , piece_id=JUKT6TKJNZ37RTL37K6K35HJ6E646C442NOYKZ34U7QHLA7CS4PA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:02.786963028 (190h20m57.21804316s) [noise=true , address=104.36.82.16:4293 , piece_id=GT3A4ZWL2K3D5IUVUU2QARQK2GP2AQU4MJXE2KWTIBV53VHL2D5A, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:48384->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:02.787060524 (190h20m57.218140656s) [noise=true , address=45.91.116.14:4189 , piece_id=5O6WXL32AWIAXB6M5MTURPI4VF22IEUITZSUNCXQ6S2RS73GTZAQ, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:46954->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:46954->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46954->45.91.116.14:4189: use of closed network connection Aug 16 15:47:02.787101383 (190h20m57.218181514s) [noise=true , address=104.36.82.8:4969 , piece_id=7M24H4PJJU5B6EUZE2F7MTW745C7VMKNGX43AD5TMKQLNNBQDUFA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:39050->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:02.787130248 (190h20m57.218210379s) [noise=true , address=77.90.31.10:4518 , piece_id=7QFOYXKKIRRPLXWOMZTJLEOTHHOEQ5GRDFSPHLWNHO7CJ6FA5FDA, seg_pos={0 0}, upload=198023]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:59534->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:02.787153102 (190h20m57.218233233s) [seg_pos={0 0}, upload=198023]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 16 15:47:03.021728667 (190h20m57.452808798s) [upload=198023]: Done uploading segments... Aug 16 15:47:03.021753324 (190h20m57.452833455s) [upload=198023]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.236706499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [.....X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXXX] duration: 1m0.507375354s FAIL (2025-08-16 15:47:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Aug 16 15:46:02.086653466 (190h19m56.517733597s) [upload=198014]: Starting upload Aug 16 15:46:02.086699654 (190h19m56.517779785s) [upload=198014]: Done starting upload Aug 16 15:46:02.086767353 (190h19m56.517847484s) [upload=198014]: Uploading segments... Aug 16 15:46:02.086874919 (190h19m56.51795505s) [upload=198014]: Got next segment. Inline: false Aug 16 15:46:02.086899336 (190h19m56.517979467s) [upload=198014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:02.086903594 (190h19m56.517983725s) [upload=198014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:46:02.573269683 (190h19m57.004349844s) [seg_pos={0 0}, upload=198014]: Begin upload segment... Aug 16 15:46:02.573341731 (190h19m57.004421862s) [seg_pos={0 0}, upload=198014]: Done begin upload segment. Aug 16 15:46:02.573364104 (190h19m57.004444235s) [noise=true , address=77.90.31.14:4690 , piece_id=FLSIQNHA4NBNQ3NTLWUKNH6IB6HK4GGS75J5ELUGE6LNTFEOUHWA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.573398389 (190h19m57.00447852s) [noise=true , address=77.90.31.18:4813 , piece_id=XLLVAB65CYQZQZJHNDJUIMM7FWDENILFGECIJRIOVVSFPNR3XBDA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.573715606 (190h19m57.004795737s) [noise=true , address=45.91.116.13:4176 , piece_id=PFYKY3TJ6LISUK47WNQZSIFWIM3WZVCIKIRP56CSN77R7CHUC3HQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.573852788 (190h19m57.004932919s) [noise=true , address=104.36.82.35:4602 , piece_id=WABERD2Z5FEBR3RL3PP2NORLDEFWMWN5GID3ZA67ZSPKZPUEIKUA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.574066337 (190h19m57.005146468s) [noise=true , address=79.127.201.215:4349 , piece_id=AGBNRMSNESUZOYQYSL5C3ECY6X7RJNOH7V6WWLJXOKT76B6TYUEA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.574351904 (190h19m57.005432035s) [noise=true , address=104.36.82.4:4806 , piece_id=DXIAKSI7BVP2CTLOJYS3IKK4XBYFJXPMWEJOEY7KLAVV5FUVIYEQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.574615107 (190h19m57.005695238s) [noise=true , address=77.90.31.13:4647 , piece_id=QIRAN5P2T4HMRZXSUZPCZUSMS4FKKZ7JL32SBOA3H62HRFEBWFVA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.574824408 (190h19m57.005904539s) [noise=true , address=104.36.82.5:4845 , piece_id=AYULAPNZ3WN6JCLW3POTFNATWDRXMVZIQTZ2A34GVPYLZZ3OB5XQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.575022256 (190h19m57.006102387s) [noise=true , address=104.36.82.50:4046 , piece_id=GBHETT4TKAB2LS6YRIYGCSTZE53BMHBSB2S73U2523N2H4Z23CQQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.575227580 (190h19m57.006307711s) [noise=true , address=77.90.31.12:4600 , piece_id=INTBABICJURAGFDST2HEAFRLO2NKUXJRVUDZGYIO5FHGZMS6KD3Q, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.575483860 (190h19m57.006563991s) [noise=true , address=77.90.31.11:4572 , piece_id=72QZ3YAR2HZ5WJNAF6R555FOB42S6TZ27YUVFEDP7OJOARCJQORA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.575688361 (190h19m57.006768492s) [noise=true , address=77.90.31.19:4872 , piece_id=QC3BAGUV6XEJKDZLACZE2KZ6VG4VC42A4JHTHDXBOVZ4B6EXFX2A, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.575918692 (190h19m57.006998823s) [noise=true , address=104.36.82.44:4881 , piece_id=3QZY4TX6FVYDSL66ZR2CCX2RDU3G5DCLSWXBHRVPQQEG3YNXIZUQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.576177426 (190h19m57.007257558s) [noise=true , address=66.23.193.175:4563 , piece_id=EOYK7DDNGIZOGOBREAKT5OKKJSHHUFKJGH477MLKJVRNBA7FBXXA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.576496377 (190h19m57.007576498s) [noise=true , address=104.36.82.60:4361 , piece_id=WEOSY6VKZ3OPX2QX24CYSAEXS3EHI7KFHV5IQWFOVHYEL5D3KT6A, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.576702481 (190h19m57.007782612s) [noise=true , address=77.90.31.16:4762 , piece_id=CDXRGFLSOLCLIGVJDFQC6LAJZACE65WSGLSGRV64ATMI2PS2FLZQ, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.576895921 (190h19m57.007976052s) [noise=true , address=104.36.82.9:4025 , piece_id=WDWSLIBC4UTSDYU6WKAPQM6DJBQPQLZGNCMGEEVLPED3ZWACNJUA, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:02.577164295 (190h19m57.008244426s) [noise=true , address=104.36.82.23:4478 , piece_id=QYNBRTTR3ZA4GW2PTP4HI6YMY2B43ILUOPPSOTMPUOGU7CBIPX5Q, seg_pos={0 0}, upload=198014]: Uploading piece... Aug 16 15:46:03.802798662 (190h19m58.233878793s) [upload=198014]: Next returned nil segment Aug 16 15:46:03.802815184 (190h19m58.233895315s) [upload=198014]: Waiting for error group managing segments... Aug 16 15:46:05.170395490 (190h19m59.601475611s) [noise=true , address=104.36.82.50:4046 , piece_id=GBHETT4TKAB2LS6YRIYGCSTZE53BMHBSB2S73U2523N2H4Z23CQQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:08.382479690 (190h20m2.813559821s) [noise=true , address=79.127.201.215:4349 , piece_id=AGBNRMSNESUZOYQYSL5C3ECY6X7RJNOH7V6WWLJXOKT76B6TYUEA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:09.117307744 (190h20m3.548387865s) [noise=true , address=77.90.31.11:4572 , piece_id=72QZ3YAR2HZ5WJNAF6R555FOB42S6TZ27YUVFEDP7OJOARCJQORA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:09.866869098 (190h20m4.297949229s) [noise=true , address=77.90.31.12:4600 , piece_id=INTBABICJURAGFDST2HEAFRLO2NKUXJRVUDZGYIO5FHGZMS6KD3Q, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:10.300981213 (190h20m4.732061344s) [noise=true , address=77.90.31.14:4690 , piece_id=FLSIQNHA4NBNQ3NTLWUKNH6IB6HK4GGS75J5ELUGE6LNTFEOUHWA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:10.896737198 (190h20m5.327817349s) [noise=true , address=77.90.31.18:4813 , piece_id=XLLVAB65CYQZQZJHNDJUIMM7FWDENILFGECIJRIOVVSFPNR3XBDA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:11.252229854 (190h20m5.683309985s) [noise=true , address=45.91.116.13:4176 , piece_id=PFYKY3TJ6LISUK47WNQZSIFWIM3WZVCIKIRP56CSN77R7CHUC3HQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:11.498738460 (190h20m5.929818591s) [noise=true , address=66.23.193.175:4563 , piece_id=EOYK7DDNGIZOGOBREAKT5OKKJSHHUFKJGH477MLKJVRNBA7FBXXA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:11.740083487 (190h20m6.171163618s) [noise=true , address=77.90.31.16:4762 , piece_id=CDXRGFLSOLCLIGVJDFQC6LAJZACE65WSGLSGRV64ATMI2PS2FLZQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:14.779752151 (190h20m9.210832282s) [noise=true , address=77.90.31.13:4647 , piece_id=QIRAN5P2T4HMRZXSUZPCZUSMS4FKKZ7JL32SBOA3H62HRFEBWFVA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:46:43.240152034 (190h20m37.671232165s) [noise=true , address=77.90.31.19:4872 , piece_id=QC3BAGUV6XEJKDZLACZE2KZ6VG4VC42A4JHTHDXBOVZ4B6EXFX2A, seg_pos={0 0}, upload=198014]: Done uploading piece. err: <nil> Aug 16 15:47:01.407287331 (190h20m55.838367462s) [noise=true , address=104.36.82.5:4845 , piece_id=AYULAPNZ3WN6JCLW3POTFNATWDRXMVZIQTZ2A34GVPYLZZ3OB5XQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube, address:104.36.82.5:4845): protocol: rpc: dial tcp 104.36.82.5:4845: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407358889 (190h20m55.83843902s) [noise=true , address=104.36.82.60:4361 , piece_id=WEOSY6VKZ3OPX2QX24CYSAEXS3EHI7KFHV5IQWFOVHYEL5D3KT6A, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k, address:104.36.82.60:4361): protocol: write tcp 188.166.225.170:32792->104.36.82.60:4361: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407404115 (190h20m55.838484246s) [noise=true , address=104.36.82.44:4881 , piece_id=3QZY4TX6FVYDSL66ZR2CCX2RDU3G5DCLSWXBHRVPQQEG3YNXIZUQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp, address:104.36.82.44:4881): protocol: rpc: dial tcp 104.36.82.44:4881: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407435806 (190h20m55.838515937s) [noise=true , address=104.36.82.9:4025 , piece_id=WDWSLIBC4UTSDYU6WKAPQM6DJBQPQLZGNCMGEEVLPED3ZWACNJUA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: rpc: dial tcp 104.36.82.9:4025: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407482385 (190h20m55.838562516s) [noise=true , address=104.36.82.23:4478 , piece_id=QYNBRTTR3ZA4GW2PTP4HI6YMY2B43ILUOPPSOTMPUOGU7CBIPX5Q, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:12EvdsFq3xEc1mmD5y1PMsNo685MbFj1zCMby8WBvxBee4emtpR, address:104.36.82.23:4478): protocol: rpc: dial tcp 104.36.82.23:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407494288 (190h20m55.838574419s) [noise=true , address=104.36.82.35:4602 , piece_id=WABERD2Z5FEBR3RL3PP2NORLDEFWMWN5GID3ZA67ZSPKZPUEIKUA, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:12oPZgRG4QqgnQLo1fgyGYoX1zd34wsd5n7p7L3mRbPD8B4JUuB, address:104.36.82.35:4602): protocol: write tcp 188.166.225.170:42636->104.36.82.35:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:47:01.407539885 (190h20m55.838620016s) [noise=true , address=104.36.82.4:4806 , piece_id=DXIAKSI7BVP2CTLOJYS3IKK4XBYFJXPMWEJOEY7KLAVV5FUVIYEQ, seg_pos={0 0}, upload=198014]: Done uploading piece. err: ecclient: upload failed (node:12qUyweKQhw15RjewCgKi1oLyCvN3VUB1VMpeGhj2Q2hszvJBsQ, address:104.36.82.4:4806): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:47:01.407574862 (190h20m55.838654993s) [seg_pos={0 0}, upload=198014]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Aug 16 15:47:01.912241555 (190h20m56.343321696s) [upload=198014]: Done uploading segments... Aug 16 15:47:01.912269728 (190h20m56.343349859s) [upload=198014]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.507386805s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.X...X..................X.X..X.....X......................X.] duration: 2.71878717s FAIL (2025-08-16 15:46:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.718815964s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 22.483326605s FAIL (2025-08-16 15:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.48335032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.253413424s FAIL (2025-08-16 15:46:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== XQRPZVRPHCOS4FJTLIOL5IYH4ZJXQF4Q, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.071833849 (190h19m33.50291397s) [noise=true , address=104.36.82.18:4389 , piece_id=TVFYHEKPEAPVT3YYWF4JKNAKWD63ZB5FQFDXXS3URHKT2JSWKYMQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072142319 (190h19m33.50322245s) [noise=true , address=79.127.201.215:4355 , piece_id=N7XJJPYOBV7HYVLGWDVNTDVYI35GSI3CYIK24N7XKNJ7NPVEMROA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072155915 (190h19m33.503236046s) [noise=true , address=104.36.82.58:4283 , piece_id=SVRRSKFV3G443JNLYYFA24VSFUV4CAKUPM3PI6IT5D7P5PMRG6GA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072501496 (190h19m33.503581627s) [noise=true , address=104.36.82.36:4638 , piece_id=NIXDCVZXCSOQBZLYAMP63VM6NJ5DJ2NWDQYODAID23HPSRUWY6WA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072599724 (190h19m33.503679855s) [noise=true , address=66.23.193.175:4555 , piece_id=LKXZE3QFS7LWK7YGBE4RLZYFTJXBHHSWMI7QSABC6QHEMIICEAXA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072758108 (190h19m33.503838239s) [noise=true , address=45.91.116.4:4950 , piece_id=YUCLD6BZLDQNGMCZ6MW6IZV2FKCOZVFERIU6A7FFPJZBRYCAAZRQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.072964933 (190h19m33.504045064s) [noise=true , address=104.36.82.44:4870 , piece_id=T7Y7CKI7BH2GXV4QXMGA5BT4MGPH46KNGHIEXLAQWNV73YR5N4IQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073020379 (190h19m33.50410051s) [noise=true , address=156.146.43.225:4457 , piece_id=UB76FCBZQE6BAFTL4EWJIVS7Q7XV22RBP3ZQOO4XAA3MCF57K4UA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073291518 (190h19m33.504371639s) [noise=true , address=77.90.31.21:4898 , piece_id=VNYWAP6RDT6K3AA7BGB7GBZJ23LBZJSZUMHLNHIJMB5EDP7YFTLA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073303712 (190h19m33.504383843s) [noise=true , address=77.90.31.15:4714 , piece_id=Z3435ONK6PM5SWK457Q4U7YUPVRSYSA2OLBNADUQJSQINS7PJOCQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073548901 (190h19m33.504629032s) [noise=true , address=45.91.116.10:4100 , piece_id=QKFI54X7OEMU6GTYVDONXV5LWQZBK6SBURSF3CIYWNHPMDJMKXQA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073627741 (190h19m33.504707872s) [noise=true , address=104.36.82.23:4477 , piece_id=7FWQPI57QWNLTXIFK75ERP5C2A7FIQP6JFW2UBLYAZAUI6PKPSDA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073877108 (190h19m33.504957239s) [noise=true , address=104.36.82.60:4366 , piece_id=BQ27SA33IBIUVSWSG4MQPU3XBQUKZQL3ZRDLSHU5VEEL546D3PHQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.073926232 (190h19m33.505006363s) [noise=true , address=77.90.31.15:4701 , piece_id=XC7ZOW6BGDZAFED35K5XFTDLUXH3OQCY4RPYYFAAPCQK4VK6N3WA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.074146574 (190h19m33.505226705s) [noise=true , address=77.90.31.12:4599 , piece_id=KN5ZSAIMDFPYZKXPBO6HO64OVCA2K65NW3G55W7GPDOD75EEL6QQ, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:39.074219834 (190h19m33.505299965s) [noise=true , address=64.49.13.169:4706 , piece_id=J55RQ5QJHA4UL75QMG5S4VZ4DUNUFQ6R7EJVLI3IM26QL46Z47HA, seg_pos={0 0}, upload=198009]: Uploading piece... Aug 16 15:45:40.231851655 (190h19m34.662931776s) [upload=198009]: Next returned nil segment Aug 16 15:45:40.231869540 (190h19m34.662949671s) [upload=198009]: Waiting for error group managing segments... Aug 16 15:45:40.307701024 (190h19m34.738781155s) [noise=true , address=156.146.43.225:4457 , piece_id=UB76FCBZQE6BAFTL4EWJIVS7Q7XV22RBP3ZQOO4XAA3MCF57K4UA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12v4SYj7tY4VFY9LqBpQDrpSmAbFEtCpPTcasMwtcSUtiQWMnJT, address:156.146.43.225:4457): protocol: storage node overloaded, request limit: 2000 Aug 16 15:45:41.534162053 (190h19m35.965242184s) [noise=true , address=77.90.31.12:4599 , piece_id=KN5ZSAIMDFPYZKXPBO6HO64OVCA2K65NW3G55W7GPDOD75EEL6QQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:41.612547015 (190h19m36.043627136s) [noise=true , address=77.90.31.21:4898 , piece_id=VNYWAP6RDT6K3AA7BGB7GBZJ23LBZJSZUMHLNHIJMB5EDP7YFTLA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:42.123964416 (190h19m36.555044547s) [noise=true , address=77.90.31.15:4714 , piece_id=Z3435ONK6PM5SWK457Q4U7YUPVRSYSA2OLBNADUQJSQINS7PJOCQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:42.182672056 (190h19m36.613752187s) [noise=true , address=77.90.31.15:4701 , piece_id=XC7ZOW6BGDZAFED35K5XFTDLUXH3OQCY4RPYYFAAPCQK4VK6N3WA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:45.380504709 (190h19m39.81158484s) [noise=true , address=45.91.116.10:4100 , piece_id=QKFI54X7OEMU6GTYVDONXV5LWQZBK6SBURSF3CIYWNHPMDJMKXQA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:46.338160999 (190h19m40.76924113s) [noise=true , address=66.23.193.175:4555 , piece_id=LKXZE3QFS7LWK7YGBE4RLZYFTJXBHHSWMI7QSABC6QHEMIICEAXA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:49.181400505 (190h19m43.612480636s) [noise=true , address=45.91.116.4:4950 , piece_id=YUCLD6BZLDQNGMCZ6MW6IZV2FKCOZVFERIU6A7FFPJZBRYCAAZRQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:45:49.338176571 (190h19m43.769256702s) [noise=true , address=79.127.201.215:4355 , piece_id=N7XJJPYOBV7HYVLGWDVNTDVYI35GSI3CYIK24N7XKNJ7NPVEMROA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: <nil> Aug 16 15:46:37.980917976 (190h20m32.411998107s) [noise=true , address=64.49.13.169:4706 , piece_id=J55RQ5QJHA4UL75QMG5S4VZ4DUNUFQ6R7EJVLI3IM26QL46Z47HA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:46:37.981028447 (190h20m32.412108608s) [noise=true , address=104.36.82.58:4283 , piece_id=SVRRSKFV3G443JNLYYFA24VSFUV4CAKUPM3PI6IT5D7P5PMRG6GA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981097048 (190h20m32.412177189s) [noise=true , address=104.36.82.44:4870 , piece_id=T7Y7CKI7BH2GXV4QXMGA5BT4MGPH46KNGHIEXLAQWNV73YR5N4IQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981165650 (190h20m32.412245811s) [noise=true , address=104.36.82.23:4477 , piece_id=7FWQPI57QWNLTXIFK75ERP5C2A7FIQP6JFW2UBLYAZAUI6PKPSDA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf, address:104.36.82.23:4477): protocol: write tcp 188.166.225.170:56262->104.36.82.23:4477: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981244110 (190h20m32.412324231s) [noise=true , address=104.36.82.36:4638 , piece_id=NIXDCVZXCSOQBZLYAMP63VM6NJ5DJ2NWDQYODAID23HPSRUWY6WA, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:1poiPDhUGb5ZGnZnZcRViY45RefT9anD83YFcqZVZspnnJ9fN5, address:104.36.82.36:4638): protocol: rpc: dial tcp 104.36.82.36:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981310016 (190h20m32.412390147s) [noise=true , address=104.36.82.5:4862 , piece_id=AYJGRS6IXRKNDEBBQCYJZ4XF444NCCLYRFL4PLP42Y7BAXCPBA5Q, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12sUT7LBYFqWMUhx7MMjEmyiFDnbJnjSVndNbTaHioaVLLSXYue, address:104.36.82.5:4862): protocol: rpc: dial tcp 104.36.82.5:4862: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981368558 (190h20m32.412448689s) [noise=true , address=104.36.82.60:4366 , piece_id=BQ27SA33IBIUVSWSG4MQPU3XBQUKZQL3ZRDLSHU5VEEL546D3PHQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV, address:104.36.82.60:4366): protocol: rpc: dial tcp 104.36.82.60:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:37.981437209 (190h20m32.41251734s) [noise=true , address=104.36.82.2:4752 , piece_id=IQZK2FYCX5Y6EHUFWKE4XQRPZVRPHCOS4FJTLIOL5IYH4ZJXQF4Q, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:12oRYWQwoe5JtDCqfSShqtyRWVQLjL4UtbokBgNpDAocBKv1vqL, address:104.36.82.2:4752): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:46:37.982190911 (190h20m32.413271032s) [noise=true , address=104.36.82.18:4389 , piece_id=TVFYHEKPEAPVT3YYWF4JKNAKWD63ZB5FQFDXXS3URHKT2JSWKYMQ, seg_pos={0 0}, upload=198009]: Done uploading piece. err: ecclient: upload failed (node:1mMqCXPNzFx4cnu8Nb3gNKcJFJ4tGJjnkwY2EVPjUBnwNVcM9R, address:104.36.82.18:4389): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:46:37.982209126 (190h20m32.413289247s) [seg_pos={0 0}, upload=198009]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:46:38.231913999 (190h20m32.66299413s) [upload=198009]: Done uploading segments... Aug 16 15:46:38.231939428 (190h20m32.663019559s) [upload=198009]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.253431559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [................................X......X..XXXXXXXXXX.XX.XXXX] duration: 30.232586136s FAIL (2025-08-16 15:46:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:45:54.503040312 (190h19m48.934120443s) [upload=198012]: Starting upload Aug 16 15:45:54.503098414 (190h19m48.934178535s) [upload=198012]: Done starting upload Aug 16 15:45:54.503173617 (190h19m48.934253748s) [upload=198012]: Uploading segments... Aug 16 15:45:54.503270483 (190h19m48.934350614s) [upload=198012]: Got next segment. Inline: false Aug 16 15:45:54.503296352 (190h19m48.934376474s) [upload=198012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:45:54.503301893 (190h19m48.934382014s) [upload=198012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:45:54.740021620 (190h19m49.171101761s) [seg_pos={0 0}, upload=198012]: Begin upload segment... Aug 16 15:45:54.740082417 (190h19m49.171162548s) [seg_pos={0 0}, upload=198012]: Done begin upload segment. Aug 16 15:45:54.740103717 (190h19m49.171183868s) [noise=true , address=77.90.31.13:4627 , piece_id=ELTAZLG33MH3G3CEXLN6GRSE4PTP7GZM3ZIUDBTYFLJLVCKPTPSA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.740118355 (190h19m49.171198486s) [noise=true , address=45.91.116.14:4191 , piece_id=JCVYWQ5QTGIAUXCHWDEFAX7CFRAMV6R65ZVJZ4YQNHCA2PDUUA4A, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.740187787 (190h19m49.171267969s) [noise=true , address=104.36.82.15:4258 , piece_id=GDTFCH423XERCYKZQIOO4GCNUYCLK5ADCWF3FYEHDCYC6I2ZETAQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.740912284 (190h19m49.171992435s) [noise=true , address=104.36.82.12:4150 , piece_id=NCT4HGLXAJLXPRHXFJF45ZVRCO3OUXL2URYHTXJFHEYV5TJEOBHA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.740925729 (190h19m49.172005871s) [noise=true , address=104.36.82.9:4006 , piece_id=RIPWDKMVZWL2DMJCUOZ7GUE2MUQKZPF72K5FB2ICRKM7N5UCBBXA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.741490260 (190h19m49.172570401s) [noise=true , address=104.36.82.10:4079 , piece_id=2IOA6EDUOJQM477HW22FT7UMS4CK3NID3SXKBVWYECG4CC2JMLCQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.742034942 (190h19m49.173115083s) [noise=true , address=77.90.31.14:4661 , piece_id=QX6OJAV3PGOZEEBRLAMR2QOFLJV55S2GWKZFM63PPJJNDJZCSNUA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.742516513 (190h19m49.173596644s) [noise=true , address=104.36.82.10:4077 , piece_id=5VSYLADLLXIEMTOVYQLXSJ6ZX3ZK6IU5I3AVFVALLEFEOLH5KCDQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.742919354 (190h19m49.173999485s) [noise=true , address=104.36.82.4:4804 , piece_id=A3WZF7LUDNJNCT67W4GJAHZ24DMIFRO5W4IAB56QCPYTDUKCPQFQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.743330351 (190h19m49.174410482s) [noise=true , address=104.36.82.7:4909 , piece_id=PKPWRWMJYP3C4SKWXQ7WZS4PL5K3ZNI7A4WYHKTZPNDNDNILHSDQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.743711440 (190h19m49.174791561s) [noise=true , address=77.90.31.10:4519 , piece_id=R5GSYUTXK4NW6ATONAXUK2CSTBO6EPONCIYPXXATWMQ3VKGDAAVQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.744104321 (190h19m49.175184453s) [noise=true , address=45.91.116.5:4297 , piece_id=KLJWGBIQ2TQ555HG2VFCZOK6ZMFSEGPYK4OW2YDKZ6RPKK26JJKA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.744478287 (190h19m49.175558418s) [noise=true , address=104.36.82.2:4732 , piece_id=V23BCTAW4IIGREYTCJJMDDRMZBFVSIILRZQP6W6BQMU6YQC5NUHQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.744851130 (190h19m49.175931261s) [noise=true , address=45.91.116.8:4045 , piece_id=4QPSOQ3YD2WAJJBKA37ADVMQRZ3RKJXDYRV25E6PVADXNNEMRUAQ, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.746771205 (190h19m49.177851346s) [noise=true , address=104.36.82.6:4875 , piece_id=7SMVQNZHNNWYVN4S5UUKTOIJK4NRUZKIB52JLAB52P5Y7ZGDVHSA, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:54.747099212 (190h19m49.178179343s) [noise=true , address=45.91.116.6:4974 , piece_id=F4UJ2FLU2LHAJCFW5OUUXK6COBHLZ7WU7KUPUY5EK6DQTA5ASQ2A, seg_pos={0 0}, upload=198012]: Uploading piece... Aug 16 15:45:56.502980118 (190h19m50.934060319s) [upload=198012]: Next returned nil segment Aug 16 15:45:56.503007791 (190h19m50.934087962s) [upload=198012]: Waiting for error group managing segments... Aug 16 15:45:56.862425147 (190h19m51.293505268s) [noise=true , address=104.36.82.6:4875 , piece_id=7SMVQNZHNNWYVN4S5UUKTOIJK4NRUZKIB52JLAB52P5Y7ZGDVHSA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:56.865982033 (190h19m51.297062164s) [noise=true , address=104.36.82.15:4258 , piece_id=GDTFCH423XERCYKZQIOO4GCNUYCLK5ADCWF3FYEHDCYC6I2ZETAQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:56.963894017 (190h19m51.394974148s) [noise=true , address=104.36.82.7:4909 , piece_id=PKPWRWMJYP3C4SKWXQ7WZS4PL5K3ZNI7A4WYHKTZPNDNDNILHSDQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:56.969068849 (190h19m51.40014898s) [noise=true , address=104.36.82.10:4077 , piece_id=5VSYLADLLXIEMTOVYQLXSJ6ZX3ZK6IU5I3AVFVALLEFEOLH5KCDQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:57.060126828 (190h19m51.491206959s) [noise=true , address=104.36.82.10:4079 , piece_id=2IOA6EDUOJQM477HW22FT7UMS4CK3NID3SXKBVWYECG4CC2JMLCQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:58.829696026 (190h19m53.260776147s) [noise=true , address=45.91.116.8:4045 , piece_id=4QPSOQ3YD2WAJJBKA37ADVMQRZ3RKJXDYRV25E6PVADXNNEMRUAQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:59.158228607 (190h19m53.589308738s) [noise=true , address=45.91.116.5:4297 , piece_id=KLJWGBIQ2TQ555HG2VFCZOK6ZMFSEGPYK4OW2YDKZ6RPKK26JJKA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:45:59.351644728 (190h19m53.782724849s) [noise=true , address=77.90.31.13:4627 , piece_id=ELTAZLG33MH3G3CEXLN6GRSE4PTP7GZM3ZIUDBTYFLJLVCKPTPSA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:46:00.871225770 (190h19m55.302305901s) [noise=true , address=77.90.31.14:4661 , piece_id=QX6OJAV3PGOZEEBRLAMR2QOFLJV55S2GWKZFM63PPJJNDJZCSNUA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:46:04.147742385 (190h19m58.578822516s) [noise=true , address=45.91.116.6:4974 , piece_id=F4UJ2FLU2LHAJCFW5OUUXK6COBHLZ7WU7KUPUY5EK6DQTA5ASQ2A, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:46:04.791078458 (190h19m59.222158589s) [noise=true , address=104.36.82.12:4150 , piece_id=NCT4HGLXAJLXPRHXFJF45ZVRCO3OUXL2URYHTXJFHEYV5TJEOBHA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:46:07.140893972 (190h20m1.571974103s) [noise=true , address=45.91.116.14:4191 , piece_id=JCVYWQ5QTGIAUXCHWDEFAX7CFRAMV6R65ZVJZ4YQNHCA2PDUUA4A, seg_pos={0 0}, upload=198012]: Done uploading piece. err: <nil> Aug 16 15:46:23.855740274 (190h20m18.286820405s) [noise=true , address=104.36.82.9:4006 , piece_id=RIPWDKMVZWL2DMJCUOZ7GUE2MUQKZPF72K5FB2ICRKM7N5UCBBXA, seg_pos={0 0}, upload=198012]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:46942->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:23.855784838 (190h20m18.28686497s) [noise=true , address=104.36.82.2:4732 , piece_id=V23BCTAW4IIGREYTCJJMDDRMZBFVSIILRZQP6W6BQMU6YQC5NUHQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36842->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:23.855791272 (190h20m18.286871393s) [noise=true , address=77.90.31.10:4519 , piece_id=R5GSYUTXK4NW6ATONAXUK2CSTBO6EPONCIYPXXATWMQ3VKGDAAVQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:23.855813384 (190h20m18.286893515s) [noise=true , address=104.36.82.4:4804 , piece_id=A3WZF7LUDNJNCT67W4GJAHZ24DMIFRO5W4IAB56QCPYTDUKCPQFQ, seg_pos={0 0}, upload=198012]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:23.855840466 (190h20m18.286920597s) [seg_pos={0 0}, upload=198012]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:46:24.086371642 (190h20m18.517451803s) [upload=198012]: Done uploading segments... Aug 16 15:46:24.086403783 (190h20m18.517483944s) [upload=198012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232612296s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.003775921s FAIL (2025-08-16 15:46:22 UTC) orbiter warning: context deadline exceeded Duration: 7.003796961s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [..X...X..................X.X..X.....X......................X] duration: 882.734625ms FAIL (2025-08-16 15:46:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:46:08.090513456 (190h20m2.521593587s) [upload=198019]: Starting upload Aug 16 15:46:08.090588269 (190h20m2.5216684s) [upload=198019]: Done starting upload Aug 16 15:46:08.090669504 (190h20m2.521749635s) [upload=198019]: Uploading segments... Aug 16 15:46:08.090773282 (190h20m2.521853493s) [upload=198019]: Got next segment. Inline: true Aug 16 15:46:08.090805795 (190h20m2.521885976s) [upload=198019]: Waiting for error group managing segments... Aug 16 15:46:08.090863295 (190h20m2.521943426s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:46:08.090881419 (190h20m2.52196155s) [upload=198019]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:46:08.090884775 (190h20m2.521964907s) [upload=198019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:46:08.311326817 (190h20m2.742406948s) [upload=198019]: Done uploading segments... Aug 16 15:46:08.311342047 (190h20m2.742422178s) [upload=198019]: Upload finished. err: metaclient: Too Many Requests Duration:* 882.751729ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [......X......X...X.X......X..X......X..X...XXXXXXXXXXXXXXXXX] duration: 1m0.237050987s FAIL (2025-08-16 15:46:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Aug 16 15:45:01.570739498 (190h18m56.001819629s) [upload=197999]: Starting upload Aug 16 15:45:01.570819701 (190h18m56.001899852s) [upload=197999]: Done starting upload Aug 16 15:45:01.571052526 (190h18m56.002132667s) [upload=197999]: Uploading segments... Aug 16 15:45:01.571383701 (190h18m56.002463832s) [upload=197999]: Got next segment. Inline: false Aug 16 15:45:01.571425600 (190h18m56.002505742s) [upload=197999]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:45:01.571431051 (190h18m56.002511182s) [upload=197999]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:45:02.071009289 (190h18m56.50208943s) [seg_pos={0 0}, upload=197999]: Begin upload segment... Aug 16 15:45:02.071080866 (190h18m56.502160997s) [seg_pos={0 0}, upload=197999]: Done begin upload segment. Aug 16 15:45:02.071115462 (190h18m56.502195603s) [noise=true , address=104.36.82.59:4325 , piece_id=AXIXPVAPV2CUPY4JT6RFPI22KU2WOA7YS3TORDNEL2VGWCL44LCA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.071167922 (190h18m56.502248073s) [noise=true , address=104.36.82.44:4895 , piece_id=BLJMW4XNGTGCFA7ZX2BWJE2CRR6P3LXA4IF6NWDS7EG2FMM2XNTA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.071684230 (190h18m56.502764362s) [noise=true , address=104.36.82.38:4702 , piece_id=MIEVJOGN62VAF37IYCTH537ZJ4OGK7DEW64GBAV6ZNLAAYZBJKKQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.071720750 (190h18m56.502800891s) [noise=true , address=77.90.31.21:4905 , piece_id=NSDWV4BSPN4WUGGGIWG4RMSQGJOMPJKIAZAUZIU7EL5TBIRQRA3A, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.072104715 (190h18m56.503184846s) [noise=true , address=66.23.193.175:4562 , piece_id=GK6E2KEGL4DHFMIEGLLJLO4SHCDVDPZPXMXBULT5MDJJJFRB2QZA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.072186902 (190h18m56.503267043s) [noise=true , address=104.36.82.34:4579 , piece_id=SEB7F3FBAHN3CNLIW2KB7AFEUVS3ECZO2BI7CVOMNJO7CDK2HRIQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.072532714 (190h18m56.503612855s) [noise=true , address=104.36.82.18:4371 , piece_id=GX4RXLZB52OVIWVYOZKRB5GAW2VWGJ64DUV3CBTH3P3OX72Q6JJA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.072873325 (190h18m56.503953466s) [noise=true , address=64.49.13.155:4922 , piece_id=J5VW563XKSDUCEZRWO4FNTXHWOEK65VUHMGROZGCUE5ZDOF65RFA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.072923742 (190h18m56.504003883s) [noise=true , address=79.127.201.215:4356 , piece_id=4YPAYVYCRWEZHYQEF6TSH7PV3SQ4HQRZSHA44VRYML5AQAXSM5JQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.073336762 (190h18m56.504416903s) [noise=true , address=104.36.82.22:4542 , piece_id=MUT5D327CB7G2ENCEYECSKDJFYWMO6FAPYZYMBXD2426CNCPOKEQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.073350519 (190h18m56.50443066s) [noise=true , address=45.91.116.8:4051 , piece_id=OT7QSXCWXHEGCH2QW4DZIUGE4N62EI7BFJONDEBNWHCGIIPJYARQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.073723342 (190h18m56.504803483s) [noise=true , address=104.36.82.60:4350 , piece_id=PB2JJOJZHTM6262B77QUI4ULHX66RZO5RO2UBOEVI67BE7FXTSZQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.073883248 (190h18m56.504963409s) [noise=true , address=77.90.31.15:4718 , piece_id=GM7G4DO267B73YYJRFG2IR7J2KAVGZ255GYBR7C2CLKUA3QMJ7IA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.074142444 (190h18m56.505222575s) [noise=true , address=104.36.82.23:4451 , piece_id=656JELLTCVA4L3XQRFWFQHENCDT7VG5O4CFPBP5HBI4GX4K5RTRA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.074266391 (190h18m56.505346532s) [noise=true , address=104.36.82.58:4296 , piece_id=Y2O2PYSNDHIUFAVY7GWXQZRLL5WOZVO7SZYUPCACCYNISFNO3LVQ, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.074717133 (190h18m56.505797264s) [noise=true , address=77.90.31.19:4859 , piece_id=DROAG7T3SMIC2BP6E57HGZCSFETJSGQGOXJI2BBRR77VHVLI6LHA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.075243040 (190h18m56.506323181s) [noise=true , address=104.36.82.2:4756 , piece_id=2NUQUDYCAL4AB3QN6HH3SLKI5ZN4TWBHV4PH7PJVJI2GDIOYJ3JA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:02.077218289 (190h18m56.50829843s) [noise=true , address=79.127.201.214:4324 , piece_id=FITO3TC2K5X2P7ZDBISWUH5BENMKXLAK2YOIKWL7JKCZ5XURWFJA, seg_pos={0 0}, upload=197999]: Uploading piece... Aug 16 15:45:03.281144280 (190h18m57.712224411s) [upload=197999]: Next returned nil segment Aug 16 15:45:03.281162485 (190h18m57.712242616s) [upload=197999]: Waiting for error group managing segments... Aug 16 15:45:03.666360990 (190h18m58.097441121s) [noise=true , address=64.49.13.155:4922 , piece_id=J5VW563XKSDUCEZRWO4FNTXHWOEK65VUHMGROZGCUE5ZDOF65RFA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:04.076643199 (190h18m58.50772334s) [noise=true , address=77.90.31.15:4718 , piece_id=GM7G4DO267B73YYJRFG2IR7J2KAVGZ255GYBR7C2CLKUA3QMJ7IA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:04.176978029 (190h18m58.60805816s) [noise=true , address=79.127.201.214:4324 , piece_id=FITO3TC2K5X2P7ZDBISWUH5BENMKXLAK2YOIKWL7JKCZ5XURWFJA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:04.264801599 (190h18m58.69588173s) [noise=true , address=77.90.31.21:4905 , piece_id=NSDWV4BSPN4WUGGGIWG4RMSQGJOMPJKIAZAUZIU7EL5TBIRQRA3A, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:05.431316128 (190h18m59.862396259s) [noise=true , address=45.91.116.8:4051 , piece_id=OT7QSXCWXHEGCH2QW4DZIUGE4N62EI7BFJONDEBNWHCGIIPJYARQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:09.053253507 (190h19m3.484333628s) [noise=true , address=66.23.193.175:4562 , piece_id=GK6E2KEGL4DHFMIEGLLJLO4SHCDVDPZPXMXBULT5MDJJJFRB2QZA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:45:11.476243680 (190h19m5.907323811s) [noise=true , address=79.127.201.215:4356 , piece_id=4YPAYVYCRWEZHYQEF6TSH7PV3SQ4HQRZSHA44VRYML5AQAXSM5JQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: <nil> Aug 16 15:46:00.932973537 (190h19m55.364053668s) [noise=true , address=104.36.82.23:4451 , piece_id=656JELLTCVA4L3XQRFWFQHENCDT7VG5O4CFPBP5HBI4GX4K5RTRA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: rpc: dial tcp 104.36.82.23:4451: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933035515 (190h19m55.364115646s) [noise=true , address=77.90.31.19:4859 , piece_id=DROAG7T3SMIC2BP6E57HGZCSFETJSGQGOXJI2BBRR77VHVLI6LHA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: rpc: dial tcp 77.90.31.19:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933084529 (190h19m55.36416466s) [noise=true , address=104.36.82.44:4895 , piece_id=BLJMW4XNGTGCFA7ZX2BWJE2CRR6P3LXA4IF6NWDS7EG2FMM2XNTA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:104.36.82.44:4895): protocol: write tcp 188.166.225.170:34260->104.36.82.44:4895: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933124687 (190h19m55.364204818s) [noise=true , address=104.36.82.60:4350 , piece_id=PB2JJOJZHTM6262B77QUI4ULHX66RZO5RO2UBOEVI67BE7FXTSZQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12byqLui89KiVQB8FetY1aAZAX3Ziy6orZr4DeSgLFjpFQe2CH, address:104.36.82.60:4350): protocol: rpc: dial tcp 104.36.82.60:4350: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933181866 (190h19m55.364261997s) [noise=true , address=104.36.82.58:4296 , piece_id=Y2O2PYSNDHIUFAVY7GWXQZRLL5WOZVO7SZYUPCACCYNISFNO3LVQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12nctdBBFSDaAV2c5PUbZW64K3v9CH3a49NSfTh6dta4DVxtqXA, address:104.36.82.58:4296): protocol: rpc: dial tcp 104.36.82.58:4296: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933207255 (190h19m55.364287386s) [noise=true , address=104.36.82.38:4702 , piece_id=MIEVJOGN62VAF37IYCTH537ZJ4OGK7DEW64GBAV6ZNLAAYZBJKKQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:127aj2R9GJTXfH96zcHEVbcsTWMT52HWiCrLZZ8c6tZsxDe1xrj, address:104.36.82.38:4702): protocol: rpc: dial tcp 104.36.82.38:4702: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933224056 (190h19m55.364304187s) [noise=true , address=104.36.82.34:4579 , piece_id=SEB7F3FBAHN3CNLIW2KB7AFEUVS3ECZO2BI7CVOMNJO7CDK2HRIQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68, address:104.36.82.34:4579): protocol: rpc: dial tcp 104.36.82.34:4579: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933260907 (190h19m55.364341038s) [noise=true , address=104.36.82.18:4371 , piece_id=GX4RXLZB52OVIWVYOZKRB5GAW2VWGJ64DUV3CBTH3P3OX72Q6JJA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:12cSqcGZqvZqs1QKBi6WK4fUDKgvQcmTupR1C9ZkWz879XeWPEL, address:104.36.82.18:4371): protocol: rpc: dial tcp 104.36.82.18:4371: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933293810 (190h19m55.364373941s) [noise=true , address=104.36.82.22:4542 , piece_id=MUT5D327CB7G2ENCEYECSKDJFYWMO6FAPYZYMBXD2426CNCPOKEQ, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:1V1MxSKuV5ZxsXXJBppZ5qmFwJPovFp8wD1biE1sF7mWzQ9qpL, address:104.36.82.22:4542): protocol: rpc: dial tcp 104.36.82.22:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933303348 (190h19m55.364383469s) [noise=true , address=104.36.82.59:4325 , piece_id=AXIXPVAPV2CUPY4JT6RFPI22KU2WOA7YS3TORDNEL2VGWCL44LCA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw, address:104.36.82.59:4325): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:46:00.933331482 (190h19m55.364411613s) [noise=true , address=104.36.82.2:4756 , piece_id=2NUQUDYCAL4AB3QN6HH3SLKI5ZN4TWBHV4PH7PJVJI2GDIOYJ3JA, seg_pos={0 0}, upload=197999]: Done uploading piece. err: ecclient: upload failed (node:1Pow7VVb1PRDBvHjEcS4bfnh3276S313jiGawszkhaJ582pDqQ, address:104.36.82.2:4756): protocol: write tcp 188.166.225.170:47624->104.36.82.2:4756: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:46:00.933346301 (190h19m55.364426432s) [seg_pos={0 0}, upload=197999]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 16 15:46:01.168064497 (190h19m55.599144628s) [upload=197999]: Done uploading segments... Aug 16 15:46:01.168088774 (190h19m55.599168905s) [upload=197999]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.237068781s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 33.971352323s FAIL (2025-08-16 15:45:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.971363455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.................................X......X..XXXXXXXXXX.XX.XXX] duration: 30.2355071s FAIL (2025-08-16 15:45:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:45:19.739839996 (190h19m14.170920127s) [upload=198004]: Starting upload Aug 16 15:45:19.739899260 (190h19m14.170979391s) [upload=198004]: Done starting upload Aug 16 15:45:19.739988630 (190h19m14.171068761s) [upload=198004]: Uploading segments... Aug 16 15:45:19.740080426 (190h19m14.171160557s) [upload=198004]: Got next segment. Inline: false Aug 16 15:45:19.740114522 (190h19m14.171194653s) [upload=198004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:45:19.740119411 (190h19m14.171199542s) [upload=198004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:45:19.977822752 (190h19m14.408902883s) [seg_pos={0 0}, upload=198004]: Begin upload segment... Aug 16 15:45:19.977866516 (190h19m14.408946647s) [seg_pos={0 0}, upload=198004]: Done begin upload segment. Aug 16 15:45:19.977893748 (190h19m14.408973879s) [noise=true , address=45.91.116.14:4191 , piece_id=SN6QJS6BH3I2NTTB3FVSOISYUJBB2VSELCISKWSHZEOM3CIS54MA, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.977897846 (190h19m14.408977977s) [noise=true , address=104.36.82.11:4115 , piece_id=ODH5Z2QFVGP2W2LLZDLDSVCSCZKN4ZU6IM6HLLIPJTW7D3GXFXMQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.977977127 (190h19m14.409057258s) [noise=true , address=104.36.82.12:4150 , piece_id=Y2F7TA6CF6WRJMX76SZAFT2CM37QWILXJT7RTHFQ6VDNH57EFQTQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978242424 (190h19m14.409322555s) [noise=true , address=104.36.82.4:4802 , piece_id=E3Z2GEC3YXHJKEBWM6HL3T7W5ZVLJP7Q7A3H5NNFA5CN3HYHSTZQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978248988 (190h19m14.409329109s) [noise=true , address=45.91.116.7:4010 , piece_id=4SBSJCZWSVZSBPY3TGVCSSZ43LQFPY2B6CBWDICMV54Q5J6Y5QXQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978494668 (190h19m14.409574799s) [noise=true , address=104.36.82.15:4259 , piece_id=ZXGXDLF2CYQDJE6GCJVMIAAH7DB2OGUFRVZ5TS7O2FDOJ34T7H3Q, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978515146 (190h19m14.409595267s) [noise=true , address=104.36.82.5:4837 , piece_id=GOQB6IYPEVXYTJ4EAP3O7QA5BWNT3IXKQZGNJYJZ3O5DX26FXVPQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978748213 (190h19m14.409828344s) [noise=true , address=45.91.116.4:4937 , piece_id=MIFOZ5RJIOYP3TZMYSU5SWDDQQUG73WP4AZWVVVIDLJ5OV3PIAXQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978759203 (190h19m14.409839334s) [noise=true , address=77.90.31.21:4877 , piece_id=KRMWCYULEX65WSPTDQC2LIXTKHX25YYM6RVPEDYOXZKSA4JV3Y3A, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.978972572 (190h19m14.410052693s) [noise=true , address=77.90.31.14:4661 , piece_id=G7X4WEPZFIQTYEQ2R3T4KMECASUOV6GN3U3HLPIVN6B3HXYJZHOQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.979187353 (190h19m14.410267484s) [noise=true , address=104.36.82.10:4078 , piece_id=FDPF6WT7S3R33BE44HMUGJ2I32RAC4JVG3Y2QUWXHBFUD5DGOTUA, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.979407885 (190h19m14.410488016s) [noise=true , address=104.36.82.12:4151 , piece_id=U7TSJCLGIWGXWXG67EIL7M7NTE2VI44LZOIVKSFVXO2O4K3NS5TA, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.979623798 (190h19m14.410703919s) [noise=true , address=104.36.82.7:4909 , piece_id=2YJUMHTAQFYZ6EC4VDS5FPMXNHPSAVB4GSXGBAZTA7PYB55DRV6A, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.979833238 (190h19m14.410913359s) [noise=true , address=77.90.31.10:4520 , piece_id=NN5FRS2CSJL6BB665FTG54HNF3CVA3ZQN7V56FRL4JPIJK7HLSCQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.980006310 (190h19m14.411086431s) [noise=true , address=104.36.82.6:4874 , piece_id=HQ5DROGCLTC4OYZ3HLNDNTZWCCQM3ECDOZWRCF4EPF7VIJOGR7HA, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:19.984676135 (190h19m14.415756256s) [noise=true , address=104.36.82.11:4114 , piece_id=HNMHLUK5KT6KM6AAWKYW6TZUIWVOL4A2WUF3VZSPAUHUN5BBFQZQ, seg_pos={0 0}, upload=198004]: Uploading piece... Aug 16 15:45:21.742391299 (190h19m16.17347143s) [upload=198004]: Next returned nil segment Aug 16 15:45:21.742409264 (190h19m16.173489395s) [upload=198004]: Waiting for error group managing segments... Aug 16 15:45:22.035140223 (190h19m16.466220344s) [noise=true , address=104.36.82.15:4259 , piece_id=ZXGXDLF2CYQDJE6GCJVMIAAH7DB2OGUFRVZ5TS7O2FDOJ34T7H3Q, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.036761345 (190h19m16.467841476s) [noise=true , address=104.36.82.10:4078 , piece_id=FDPF6WT7S3R33BE44HMUGJ2I32RAC4JVG3Y2QUWXHBFUD5DGOTUA, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.073073885 (190h19m16.504154016s) [noise=true , address=104.36.82.6:4874 , piece_id=HQ5DROGCLTC4OYZ3HLNDNTZWCCQM3ECDOZWRCF4EPF7VIJOGR7HA, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.136476239 (190h19m16.56755636s) [noise=true , address=104.36.82.12:4151 , piece_id=U7TSJCLGIWGXWXG67EIL7M7NTE2VI44LZOIVKSFVXO2O4K3NS5TA, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.253053877 (190h19m16.684134048s) [noise=true , address=77.90.31.21:4877 , piece_id=KRMWCYULEX65WSPTDQC2LIXTKHX25YYM6RVPEDYOXZKSA4JV3Y3A, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.288945882 (190h19m16.720026013s) [noise=true , address=104.36.82.7:4909 , piece_id=2YJUMHTAQFYZ6EC4VDS5FPMXNHPSAVB4GSXGBAZTA7PYB55DRV6A, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.531959502 (190h19m16.963039623s) [noise=true , address=45.91.116.7:4010 , piece_id=4SBSJCZWSVZSBPY3TGVCSSZ43LQFPY2B6CBWDICMV54Q5J6Y5QXQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.563796078 (190h19m16.994876209s) [noise=true , address=45.91.116.14:4191 , piece_id=SN6QJS6BH3I2NTTB3FVSOISYUJBB2VSELCISKWSHZEOM3CIS54MA, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.650545443 (190h19m17.081625574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIFOZ5RJIOYP3TZMYSU5SWDDQQUG73WP4AZWVVVIDLJ5OV3PIAXQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.739750117 (190h19m17.170830248s) [noise=true , address=104.36.82.12:4150 , piece_id=Y2F7TA6CF6WRJMX76SZAFT2CM37QWILXJT7RTHFQ6VDNH57EFQTQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:22.814507957 (190h19m17.245588089s) [noise=true , address=77.90.31.14:4661 , piece_id=G7X4WEPZFIQTYEQ2R3T4KMECASUOV6GN3U3HLPIVN6B3HXYJZHOQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: <nil> Aug 16 15:45:49.090336213 (190h19m43.521416344s) [noise=true , address=77.90.31.10:4520 , piece_id=NN5FRS2CSJL6BB665FTG54HNF3CVA3ZQN7V56FRL4JPIJK7HLSCQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:49.090385276 (190h19m43.521465407s) [noise=true , address=104.36.82.5:4837 , piece_id=GOQB6IYPEVXYTJ4EAP3O7QA5BWNT3IXKQZGNJYJZ3O5DX26FXVPQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:40930->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:49.090428159 (190h19m43.52150829s) [noise=true , address=104.36.82.11:4115 , piece_id=ODH5Z2QFVGP2W2LLZDLDSVCSCZKN4ZU6IM6HLLIPJTW7D3GXFXMQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 188.166.225.170:34016->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:49.090472624 (190h19m43.521552755s) [noise=true , address=104.36.82.4:4802 , piece_id=E3Z2GEC3YXHJKEBWM6HL3T7W5ZVLJP7Q7A3H5NNFA5CN3HYHSTZQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:58922->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:49.090491750 (190h19m43.521571881s) [noise=true , address=104.36.82.11:4114 , piece_id=HNMHLUK5KT6KM6AAWKYW6TZUIWVOL4A2WUF3VZSPAUHUN5BBFQZQ, seg_pos={0 0}, upload=198004]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:49.090521707 (190h19m43.521601828s) [seg_pos={0 0}, upload=198004]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:45:49.324818754 (190h19m43.755898885s) [upload=198004]: Done uploading segments... Aug 16 15:45:49.324843000 (190h19m43.755923131s) [upload=198004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.235522218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 1m0.225993506s FAIL (2025-08-16 15:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 53 , piece_id=KHIK7USZR5MOMALHUMMZV3QUKQPBUWGYNTWAQOX4ZGFZPCVTCGQA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.600236507 (190h18m33.031316638s) [noise=true , address=104.36.82.59:4317 , piece_id=7NCUPYB6O5XBIQ3MZRHU2NHVGBLN7TAJ6OSFEB2DDE2T5PY3AZBQ, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.600590094 (190h18m33.031670225s) [noise=true , address=66.23.193.175:4558 , piece_id=RKMO6VH4VAJNRNZXWMEOQCX43VRP4CGEJDBMAQAZCZ75KJ2HTYMA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.600619219 (190h18m33.03169935s) [noise=true , address=79.127.205.235:4661 , piece_id=M66U6EUNXWQE4PQZE6MWSGER2QVX54H2B7KF6Q4Y3D4LIFXYAV4Q, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.600898454 (190h18m33.031978585s) [noise=true , address=104.36.82.44:4899 , piece_id=KWLQ4C2NGUDLKU5YAWBOM54STU534PBRQ3JRAE5VF4JY22NKPKYQ, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.600936356 (190h18m33.032016477s) [noise=true , address=104.36.82.37:4680 , piece_id=OUAML5KFGZSIOANT37BEV3QZ7F5SCOSYFAVJFLBRA2E4E2RSGO7A, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.601152650 (190h18m33.032232781s) [noise=true , address=104.36.82.5:4848 , piece_id=5OEC2SRMZLARABR2UPCLVJZXW6TG54A2LEUY57AWTCW6AONJNR5Q, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.601243003 (190h18m33.032323124s) [noise=true , address=45.91.116.7:4017 , piece_id=N2T2PWELQKABFKHF56MMHREXN5XPJPJ4F5HPHTOETWEJOZWBSQYQ, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.601373131 (190h18m33.032453263s) [noise=true , address=79.127.201.214:4322 , piece_id=JPZINR4OIA7OWQVSMWZ3SXQJYOFFGSMEPRHV3Z3KZGBJKSITULJA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.601569909 (190h18m33.03265004s) [noise=true , address=77.90.31.19:4865 , piece_id=M2M6PWRT2A3V5QAS673F3CWQP6VIYGKH2EOUJGWDLJNT67EO2GBQ, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.601925890 (190h18m33.033006021s) [noise=true , address=77.90.31.15:4726 , piece_id=2D6SDEGJGSRFYGZXNQN6GQ474RJD4PWHEDHX6THIRSSD6RYTNCTA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.602647891 (190h18m33.033728012s) [noise=true , address=77.90.31.10:4536 , piece_id=5JC7YACEKOG3MWMRUCM2SOJIMWFRWC6I57XMTS2TWQXJM2QAC3KA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.604206674 (190h18m33.035286805s) [noise=true , address=104.36.82.23:4451 , piece_id=UHWBX3JY3GN5OSBGFDSWBVARHSOMM5M2LF7WFT6VTM44XC5CPADA, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:38.604485106 (190h18m33.035565237s) [noise=true , address=77.90.31.21:4900 , piece_id=27KZYU576LML2C2V4I63EHNMOYDHW76IYDUXCAFMDOUJVBYBDRPQ, seg_pos={0 0}, upload=197991]: Uploading piece... Aug 16 15:44:39.714277733 (190h18m34.145357864s) [upload=197991]: Next returned nil segment Aug 16 15:44:39.714292922 (190h18m34.145373043s) [upload=197991]: Waiting for error group managing segments... Aug 16 15:44:40.571619654 (190h18m35.002699785s) [noise=true , address=77.90.31.15:4726 , piece_id=2D6SDEGJGSRFYGZXNQN6GQ474RJD4PWHEDHX6THIRSSD6RYTNCTA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:40.821296423 (190h18m35.252376554s) [noise=true , address=77.90.31.11:4580 , piece_id=657AW2ZMRLBR4CSS6N6M5WY355YUTCJNR6TU5FT3OZRG4MOLNOXQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:41.025134725 (190h18m35.456214856s) [noise=true , address=77.90.31.21:4900 , piece_id=27KZYU576LML2C2V4I63EHNMOYDHW76IYDUXCAFMDOUJVBYBDRPQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:41.407548351 (190h18m35.838628482s) [noise=true , address=79.127.201.215:4353 , piece_id=KHIK7USZR5MOMALHUMMZV3QUKQPBUWGYNTWAQOX4ZGFZPCVTCGQA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:41.805661918 (190h18m36.236742049s) [noise=true , address=79.127.201.214:4322 , piece_id=JPZINR4OIA7OWQVSMWZ3SXQJYOFFGSMEPRHV3Z3KZGBJKSITULJA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:41.870535997 (190h18m36.301616128s) [noise=true , address=79.127.205.235:4661 , piece_id=M66U6EUNXWQE4PQZE6MWSGER2QVX54H2B7KF6Q4Y3D4LIFXYAV4Q, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:44.269271403 (190h18m38.700351534s) [noise=true , address=45.91.116.7:4017 , piece_id=N2T2PWELQKABFKHF56MMHREXN5XPJPJ4F5HPHTOETWEJOZWBSQYQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:44:53.634260827 (190h18m48.065340988s) [noise=true , address=66.23.193.175:4558 , piece_id=RKMO6VH4VAJNRNZXWMEOQCX43VRP4CGEJDBMAQAZCZ75KJ2HTYMA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: <nil> Aug 16 15:45:37.512653187 (190h19m31.943733318s) [noise=true , address=104.36.82.19:4414 , piece_id=7LLBDFNHUDQD4EQIEEMT6JVCJHCURQJBQY3QCW2QGICNH7EAWMKQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:48768->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512705547 (190h19m31.943785678s) [noise=true , address=104.36.82.8:4003 , piece_id=CGZYRAARAMM7ILVFUZ5A57IPMU7BUZN72FJI3MOSWJ7ID2OWKQWA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: write tcp 188.166.225.170:48344->104.36.82.8:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512819776 (190h19m31.943899907s) [noise=true , address=104.36.82.59:4317 , piece_id=7NCUPYB6O5XBIQ3MZRHU2NHVGBLN7TAJ6OSFEB2DDE2T5PY3AZBQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:12qjPfeVTW7hUd6VtE8XrhPPq2CCmYxZqxsDXps1CPGHpHahNEz, address:104.36.82.59:4317): protocol: write tcp 188.166.225.170:56548->104.36.82.59:4317: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512839113 (190h19m31.943919244s) [noise=true , address=104.36.82.23:4451 , piece_id=UHWBX3JY3GN5OSBGFDSWBVARHSOMM5M2LF7WFT6VTM44XC5CPADA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:12bC4P3xBiPFkxLvavGFVbWKnbprCStL6UJgfKSA2Wd2fX7Mbfw, address:104.36.82.23:4451): protocol: write tcp 188.166.225.170:39142->104.36.82.23:4451: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512889269 (190h19m31.9439694s) [noise=true , address=77.90.31.19:4865 , piece_id=M2M6PWRT2A3V5QAS673F3CWQP6VIYGKH2EOUJGWDLJNT67EO2GBQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 188.166.225.170:47316->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512931570 (190h19m31.944011701s) [noise=true , address=104.36.82.44:4899 , piece_id=KWLQ4C2NGUDLKU5YAWBOM54STU534PBRQ3JRAE5VF4JY22NKPKYQ, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: write tcp 188.166.225.170:43698->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512953923 (190h19m31.944034054s) [noise=true , address=104.36.82.37:4680 , piece_id=OUAML5KFGZSIOANT37BEV3QZ7F5SCOSYFAVJFLBRA2E4E2RSGO7A, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: write tcp 188.166.225.170:35206->104.36.82.37:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.512968721 (190h19m31.944048852s) [noise=true , address=104.36.82.5:4848 , piece_id=5OEC2SRMZLARABR2UPCLVJZXW6TG54A2LEUY57AWTCW6AONJNR5Q, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1xvod7N8eEwunLDYWAvVUP1Cp1CdEr3quJUMG5kWyBrwzzr9vp, address:104.36.82.5:4848): protocol: write tcp 188.166.225.170:60762->104.36.82.5:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.513005481 (190h19m31.944085612s) [noise=true , address=77.90.31.10:4536 , piece_id=5JC7YACEKOG3MWMRUCM2SOJIMWFRWC6I57XMTS2TWQXJM2QAC3KA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: write tcp 188.166.225.170:41966->77.90.31.10:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.513032673 (190h19m31.944112804s) [noise=true , address=104.36.82.58:4293 , piece_id=H4YC7YBIZ2L2M7CXSK3IIRGAUQZX5W4WTG327POBQKSH7V3SPFYA, seg_pos={0 0}, upload=197991]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 188.166.225.170:37368->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:45:37.513060186 (190h19m31.944140317s) [seg_pos={0 0}, upload=197991]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 16 15:45:37.736848270 (190h19m32.167928391s) [upload=197991]: Done uploading segments... Aug 16 15:45:37.736872126 (190h19m32.167952257s) [upload=197991]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.226004817s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)