Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.017356472s FAIL (2025-07-26 22:50:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.017380376s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X.X...X..........................................X.......] duration: 5.739432133s FAIL (2025-07-26 22:49:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.739456599s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........................X...X...............X..XX...X.X.X.] duration: 21.268865756s FAIL (2025-07-26 22:49:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.268887869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.........................X..........X.X........X.X........] duration: 5.693865397s FAIL (2025-07-26 22:49:41 UTC) orbiter warning: uplink: too many requests Duration: 5.693904261s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXX] duration: 1m0.226767779s FAIL (2025-07-26 22:49:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== .548759216s) [upload=56817]: Done starting upload Jul 26 22:48:35.826303120 (59h12m45.548866161s) [upload=56817]: Uploading segments... Jul 26 22:48:35.826402511 (59h12m45.548965551s) [upload=56817]: Got next segment. Inline: false Jul 26 22:48:35.826445704 (59h12m45.549008744s) [upload=56817]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:48:35.826450904 (59h12m45.549013944s) [upload=56817]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:48:36.267403413 (59h12m45.989966494s) [seg_pos={0 0}, upload=56817]: Begin upload segment... Jul 26 22:48:36.267465262 (59h12m45.990028292s) [seg_pos={0 0}, upload=56817]: Done begin upload segment. Jul 26 22:48:36.267501060 (59h12m45.99006416s) [noise=true , address=79.127.201.215:4348 , piece_id=QKKZAHZ6LGZLIGTZNM6GBRPPKRMUFH6MLOI42R7G47RLOQ5C4OZA, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.267935491 (59h12m45.990498591s) [noise=true , address=79.127.219.47:4061 , piece_id=LHZDGPJ7DCT4OOGGEFNPMDHOSYDUS2FNVXCY5FUMSUXJ2Y2F5HPQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.268476456 (59h12m45.991039537s) [noise=true , address=79.127.220.101:4048 , piece_id=XOK2Q57V4BWDBV6MCEPZAH5EWXRQLI5Q55LSZSR55IGJXTTX3QPA, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.268954070 (59h12m45.99151715s) [noise=true , address=79.127.205.245:4221 , piece_id=VPS6VLUSOBRORYNOOZQEEPTBUBJQNJHCW3ELZJQGZVPQGFY2HX4A, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.269129155 (59h12m45.991692245s) [noise=true , address=45.91.116.10:4100 , piece_id=44TDLVX2FSAAEG7SPRVLMHEV3DT2OY7347GQ3TLX3DBAMKKNPCYA, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.269724233 (59h12m45.992287274s) [noise=true , address=45.91.116.12:4229 , piece_id=KUXDD4JRTFZ4QHWCQ6YXSWUZEG4CBAJ42KMWFFPQKTNUG6W4Y2DQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.269734974 (59h12m45.992298014s) [noise=true , address=45.91.116.13:4175 , piece_id=7GEPGMNPPPZ7ECLVUP7ZGRZDX7KLKBHUSUVVIHGWJTGA3EV2YAWQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.270298653 (59h12m45.992861693s) [noise=true , address=79.127.205.235:4657 , piece_id=SEPNM67SGNPD3GNZMFELT2IXUHGZJYKXDH66P3RAXIWGFKTJWQCA, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.270415306 (59h12m45.992978356s) [noise=true , address=156.146.43.225:4448 , piece_id=MZQQXT2GYMYBZQWQ2UDKXLAX6UNNL4BHTHQYW47RDPSBT7HBXD2Q, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.270752240 (59h12m45.99331529s) [noise=true , address=64.49.13.155:4916 , piece_id=HAL4R52U3JTVQ3QVJGZMPV4TQ42TFZR5MWA55PS2G2DWDSOIISZQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.270942133 (59h12m45.993505183s) [noise=true , address=79.127.220.97:4361 , piece_id=QU2RQ2DGD54TZIDY4XODQ6DTRRDKUL5IAIAD5GPCTIB4RVKTC7BQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.271182233 (59h12m45.993745293s) [noise=true , address=79.127.219.38:4018 , piece_id=5AJ4GOHRAGO5DIGX4KXCQJHGY3A7MMAAGUVXJJ24E2XP437IQW7A, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.271473641 (59h12m45.994036701s) [noise=true , address=45.91.116.14:4206 , piece_id=MSIM7VTYUROVO3NOXRRF2TECS77DCWPVBH7G4UOOYIQMJHGLUM4A, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.271594211 (59h12m45.994157271s) [noise=true , address=64.49.13.140:4508 , piece_id=6U3LIEKZ56ZUILT2ESUAAK7WOLHZVBO2AJXJGSUCZJYWYJ5HJAXQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.271899335 (59h12m45.994462385s) [noise=true , address=77.90.31.18:4821 , piece_id=3MOETZT7IAHY6MQ7NEW6WJ7WOXDT42HB5FS6S477JT77YP5MGI4A, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.271971544 (59h12m45.994534584s) [noise=true , address=77.90.31.12:4596 , piece_id=O6H766AK3W2VAOAGI36MOMYAD3OF6WB2YSX3GLUSADZL7ECY6ELQ, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.272317264 (59h12m45.994880294s) [noise=true , address=77.90.31.13:4659 , piece_id=3RHQOSKXQ522OOMIGIE3P4CTLI7HLYJIAS5D2HFSBWHIPDL6YX2Q, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:36.272691711 (59h12m45.995254751s) [noise=true , address=64.49.13.140:4515 , piece_id=S6PBILJBICWMYD3HDJ35ZU7IAA7EUWZN45TQJCHX65J3WRGVSC3A, seg_pos={0 0}, upload=56817]: Uploading piece... Jul 26 22:48:37.258638462 (59h12m46.981201492s) [noise=true , address=64.49.13.140:4508 , piece_id=6U3LIEKZ56ZUILT2ESUAAK7WOLHZVBO2AJXJGSUCZJYWYJ5HJAXQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:37.366897352 (59h12m47.089460382s) [upload=56817]: Next returned nil segment Jul 26 22:48:37.366912902 (59h12m47.089475942s) [upload=56817]: Waiting for error group managing segments... Jul 26 22:48:37.644065636 (59h12m47.366628676s) [noise=true , address=79.127.205.235:4657 , piece_id=SEPNM67SGNPD3GNZMFELT2IXUHGZJYKXDH66P3RAXIWGFKTJWQCA, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:37.874989380 (59h12m47.59755241s) [noise=true , address=64.49.13.140:4515 , piece_id=S6PBILJBICWMYD3HDJ35ZU7IAA7EUWZN45TQJCHX65J3WRGVSC3A, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:38.020085340 (59h12m47.74264838s) [noise=true , address=79.127.205.245:4221 , piece_id=VPS6VLUSOBRORYNOOZQEEPTBUBJQNJHCW3ELZJQGZVPQGFY2HX4A, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:38.334232084 (59h12m48.056795124s) [noise=true , address=79.127.219.38:4018 , piece_id=5AJ4GOHRAGO5DIGX4KXCQJHGY3A7MMAAGUVXJJ24E2XP437IQW7A, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:39.234498877 (59h12m48.957061917s) [noise=true , address=45.91.116.14:4206 , piece_id=MSIM7VTYUROVO3NOXRRF2TECS77DCWPVBH7G4UOOYIQMJHGLUM4A, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:39.303349248 (59h12m49.025912278s) [noise=true , address=77.90.31.13:4659 , piece_id=3RHQOSKXQ522OOMIGIE3P4CTLI7HLYJIAS5D2HFSBWHIPDL6YX2Q, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:40.744512420 (59h12m50.46707546s) [noise=true , address=156.146.43.225:4448 , piece_id=MZQQXT2GYMYBZQWQ2UDKXLAX6UNNL4BHTHQYW47RDPSBT7HBXD2Q, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:41.173340341 (59h12m50.895903381s) [noise=true , address=79.127.220.97:4361 , piece_id=QU2RQ2DGD54TZIDY4XODQ6DTRRDKUL5IAIAD5GPCTIB4RVKTC7BQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:41.869277289 (59h12m51.591840319s) [noise=true , address=79.127.219.47:4061 , piece_id=LHZDGPJ7DCT4OOGGEFNPMDHOSYDUS2FNVXCY5FUMSUXJ2Y2F5HPQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:43.163435368 (59h12m52.885998408s) [noise=true , address=79.127.220.101:4048 , piece_id=XOK2Q57V4BWDBV6MCEPZAH5EWXRQLI5Q55LSZSR55IGJXTTX3QPA, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:43.309460065 (59h12m53.032023105s) [noise=true , address=45.91.116.10:4100 , piece_id=44TDLVX2FSAAEG7SPRVLMHEV3DT2OY7347GQ3TLX3DBAMKKNPCYA, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:43.392407728 (59h12m53.114970768s) [noise=true , address=64.49.13.155:4916 , piece_id=HAL4R52U3JTVQ3QVJGZMPV4TQ42TFZR5MWA55PS2G2DWDSOIISZQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:12MD2UDNXEmeGra9fvhEb8eGL2R6vGLHtSKLpSyX3H3tZCe1r5B, address:64.49.13.155:4916): protocol: storage node overloaded, request limit: 2000 Jul 26 22:48:47.361004862 (59h12m57.083567892s) [noise=true , address=45.91.116.12:4229 , piece_id=KUXDD4JRTFZ4QHWCQ6YXSWUZEG4CBAJ42KMWFFPQKTNUG6W4Y2DQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:51.571715399 (59h13m1.29427844s) [noise=true , address=45.91.116.13:4175 , piece_id=7GEPGMNPPPZ7ECLVUP7ZGRZDX7KLKBHUSUVVIHGWJTGA3EV2YAWQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:48:54.285023259 (59h13m4.007586299s) [noise=true , address=79.127.201.215:4348 , piece_id=QKKZAHZ6LGZLIGTZNM6GBRPPKRMUFH6MLOI42R7G47RLOQ5C4OZA, seg_pos={0 0}, upload=56817]: Done uploading piece. err: <nil> Jul 26 22:49:18.090441237 (59h13m27.813004277s) [noise=true , address=77.90.31.12:4596 , piece_id=O6H766AK3W2VAOAGI36MOMYAD3OF6WB2YSX3GLUSADZL7ECY6ELQ, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:1xgzDjjKfA3YT4tjGq6Esw9xfhA4xWMWnDbyAmNFUcP85qyFGf, address:77.90.31.12:4596): protocol: storage node overloaded, request limit: 2000 Jul 26 22:49:35.188390245 (59h13m44.910953275s) [noise=true , address=77.90.31.18:4821 , piece_id=3MOETZT7IAHY6MQ7NEW6WJ7WOXDT42HB5FS6S477JT77YP5MGI4A, seg_pos={0 0}, upload=56817]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:49:35.188429491 (59h13m44.910992531s) [seg_pos={0 0}, upload=56817]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:49:35.413605757 (59h13m45.136168797s) [upload=56817]: Done uploading segments... Jul 26 22:49:35.413632158 (59h13m45.136195198s) [upload=56817]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.226787826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.1935764s FAIL (2025-07-26 22:49:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.19359713s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........................X...X...............X..XX...X.X.X] duration: 1m0.001030707s FAIL (2025-07-26 22:49:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001049623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.........................X..........X.X........X.X.......] duration: 5.751481623s FAIL (2025-07-26 22:48:56 UTC) orbiter warning: uplink: too many requests Duration: 5.751505991s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX..X.X...X..........................................X......] duration: 5.749514994s FAIL (2025-07-26 22:48:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749535434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXX] duration: 1m0.239848187s FAIL (2025-07-26 22:48:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== t segment. Inline: false Jul 26 22:47:35.344218284 (59h11m45.066781324s) [upload=56802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:47:35.344222432 (59h11m45.066785472s) [upload=56802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:47:35.786474629 (59h11m45.509037669s) [seg_pos={0 0}, upload=56802]: Begin upload segment... Jul 26 22:47:35.786533161 (59h11m45.509096201s) [seg_pos={0 0}, upload=56802]: Done begin upload segment. Jul 26 22:47:35.786554721 (59h11m45.509117761s) [noise=true , address=77.90.31.17:4774 , piece_id=GLBO4C7AP7ILBFUVPUBVWQSYH4JBOXE26FUEVHWQR6K5QQ3IMCPQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.786658119 (59h11m45.509221159s) [noise=true , address=77.90.31.16:4760 , piece_id=VWFRHNF7ZAEKRJBNOSHE5HK33QR5CL3ZCTFSA5BNQ3UKXZQPIFEQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.786901455 (59h11m45.509464495s) [noise=true , address=79.127.220.101:4048 , piece_id=2RFICDZMQZMRP3CXNTPLYFPHHRCMPSEWLMJYKT4WM36LIFPFQGUQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787065318 (59h11m45.509628358s) [noise=true , address=64.49.13.170:4721 , piece_id=5KVJFIZ7PIQVNTSMOUHY5SEQGZNDI2DTGZ4DKGQFJO4VRA223JZA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787151784 (59h11m45.509714824s) [noise=true , address=77.90.31.21:4912 , piece_id=TXFUEGZW5TXSG5UUSPT5AB5TDL4PVNW4XSM3S3N365DUIVYS2MUA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787331718 (59h11m45.509894759s) [noise=true , address=79.127.205.233:4646 , piece_id=JBTLFAZGQOPRZINHP75UM2FZA7I7PTUGL4QOQLN7VWO5BOVPKPHQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787396943 (59h11m45.509959983s) [noise=true , address=156.146.43.225:4448 , piece_id=IAY2GTOZRHABYDTENT6W5Z2GL3BBMOKZDIGSSTXVVAPS5NSXULXA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787579943 (59h11m45.510142983s) [noise=true , address=79.127.205.235:4657 , piece_id=XA7CCZK6VTNCSHNUZPPMJR6KBWHBQ6SFVNDJ5VI2K56NM5TXGNHQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787648644 (59h11m45.510211684s) [noise=true , address=79.127.219.38:4005 , piece_id=UQLMCRXN6YE6DAR4FTKONAONP3RP4GAIXSDXDVEA6TSS6FSD5FOA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787836604 (59h11m45.510399634s) [noise=true , address=79.127.219.51:4210 , piece_id=G64CKWBKTASLB5ZLAAPQ33CL5DZZ7BQS5OPBZCURJ6HOXZK5II4A, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.787890297 (59h11m45.510453327s) [noise=true , address=64.49.13.138:4482 , piece_id=YFP5JYS3LCWKO3DJMWWYXC34DI7IWP7ZAMVLD2XKY3WVMS3GZQHA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788081603 (59h11m45.510644643s) [noise=true , address=77.90.31.11:4575 , piece_id=LRWKPOXFYZPMI3NFOVCUKBL73NQGSXGPGUJEUMPJAQJE3MUGNYVA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788116700 (59h11m45.51067974s) [noise=true , address=79.127.219.51:4212 , piece_id=NLMKS5SHBTB2PLBYXDP7GDYZUT55M5QEPK6YLH6YD7H7XL6OBQXQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788285603 (59h11m45.510848643s) [noise=true , address=79.127.201.215:4344 , piece_id=LHBJS6DRCFSSQ4ZOWMJCVGWGPAKGCO3AEKQZ6L77WROVADNQBPQQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788304018 (59h11m45.510867058s) [noise=true , address=79.127.201.215:4357 , piece_id=GRB42DDDVFXKAEDS5RDKUQW4PUZBM7IEK7A2MKLR6U6HHLWEV6OQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788517948 (59h11m45.511080988s) [noise=true , address=79.127.219.38:4001 , piece_id=KPS3SXIMMZDDHEIYJOWYD4GC2IHGSYPVDSNNJXB47UGFULS6KJFQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788547244 (59h11m45.511110284s) [noise=true , address=79.127.205.234:4605 , piece_id=S7FEJ4DSTBKSCF35LDUNE6KWQB3ULLNF23FUZQ6MELGPELISNOSQ, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:35.788766323 (59h11m45.511329363s) [noise=true , address=79.127.219.47:4069 , piece_id=662UEIOZOG6VOHTNUTKPCZWNPGGHCCY2Y5FK7HQLG5QHOCEOLYWA, seg_pos={0 0}, upload=56802]: Uploading piece... Jul 26 22:47:36.690202801 (59h11m46.412765842s) [noise=true , address=64.49.13.170:4721 , piece_id=5KVJFIZ7PIQVNTSMOUHY5SEQGZNDI2DTGZ4DKGQFJO4VRA223JZA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:47:36.792739414 (59h11m46.515302454s) [noise=true , address=156.146.43.225:4448 , piece_id=IAY2GTOZRHABYDTENT6W5Z2GL3BBMOKZDIGSSTXVVAPS5NSXULXA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:47:36.817254037 (59h11m46.539817077s) [noise=true , address=79.127.219.47:4069 , piece_id=662UEIOZOG6VOHTNUTKPCZWNPGGHCCY2Y5FK7HQLG5QHOCEOLYWA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:47:36.894915536 (59h11m46.617478576s) [upload=56802]: Next returned nil segment Jul 26 22:47:36.894933390 (59h11m46.61749643s) [upload=56802]: Waiting for error group managing segments... Jul 26 22:47:36.906649927 (59h11m46.629212967s) [noise=true , address=79.127.220.101:4048 , piece_id=2RFICDZMQZMRP3CXNTPLYFPHHRCMPSEWLMJYKT4WM36LIFPFQGUQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:47:36.935074372 (59h11m46.657637412s) [noise=true , address=79.127.219.51:4210 , piece_id=G64CKWBKTASLB5ZLAAPQ33CL5DZZ7BQS5OPBZCURJ6HOXZK5II4A, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:47:37.180929062 (59h11m46.903492102s) [noise=true , address=79.127.205.235:4657 , piece_id=XA7CCZK6VTNCSHNUZPPMJR6KBWHBQ6SFVNDJ5VI2K56NM5TXGNHQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:37.193478924 (59h11m46.916041964s) [noise=true , address=79.127.205.233:4646 , piece_id=JBTLFAZGQOPRZINHP75UM2FZA7I7PTUGL4QOQLN7VWO5BOVPKPHQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:37.201075518 (59h11m46.923638559s) [noise=true , address=79.127.205.234:4605 , piece_id=S7FEJ4DSTBKSCF35LDUNE6KWQB3ULLNF23FUZQ6MELGPELISNOSQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:37.685970974 (59h11m47.408534004s) [noise=true , address=77.90.31.11:4575 , piece_id=LRWKPOXFYZPMI3NFOVCUKBL73NQGSXGPGUJEUMPJAQJE3MUGNYVA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:37.695553778 (59h11m47.418116818s) [noise=true , address=79.127.219.38:4001 , piece_id=KPS3SXIMMZDDHEIYJOWYD4GC2IHGSYPVDSNNJXB47UGFULS6KJFQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:37.696114370 (59h11m47.41867741s) [noise=true , address=77.90.31.21:4912 , piece_id=TXFUEGZW5TXSG5UUSPT5AB5TDL4PVNW4XSM3S3N365DUIVYS2MUA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:38.465105142 (59h11m48.187668212s) [noise=true , address=79.127.219.38:4005 , piece_id=UQLMCRXN6YE6DAR4FTKONAONP3RP4GAIXSDXDVEA6TSS6FSD5FOA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:40.890109978 (59h11m50.612673018s) [noise=true , address=79.127.219.51:4212 , piece_id=NLMKS5SHBTB2PLBYXDP7GDYZUT55M5QEPK6YLH6YD7H7XL6OBQXQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:47:42.667070538 (59h11m52.389633578s) [noise=true , address=64.49.13.138:4482 , piece_id=YFP5JYS3LCWKO3DJMWWYXC34DI7IWP7ZAMVLD2XKY3WVMS3GZQHA, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:48:07.131457969 (59h12m16.854021009s) [noise=true , address=79.127.201.215:4357 , piece_id=GRB42DDDVFXKAEDS5RDKUQW4PUZBM7IEK7A2MKLR6U6HHLWEV6OQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:48:33.274703914 (59h12m42.997266954s) [noise=true , address=79.127.201.215:4344 , piece_id=LHBJS6DRCFSSQ4ZOWMJCVGWGPAKGCO3AEKQZ6L77WROVADNQBPQQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: <nil> Jul 26 22:48:34.705775407 (59h12m44.428338437s) [noise=true , address=77.90.31.16:4760 , piece_id=VWFRHNF7ZAEKRJBNOSHE5HK33QR5CL3ZCTFSA5BNQ3UKXZQPIFEQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, address:77.90.31.16:4760): protocol: write tcp 188.166.225.170:46558->77.90.31.16:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:48:34.705787370 (59h12m44.42835041s) [noise=true , address=77.90.31.17:4774 , piece_id=GLBO4C7AP7ILBFUVPUBVWQSYH4JBOXE26FUEVHWQR6K5QQ3IMCPQ, seg_pos={0 0}, upload=56802]: Done uploading piece. err: ecclient: upload failed (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y, address:77.90.31.17:4774): protocol: write tcp 188.166.225.170:59630->77.90.31.17:4774: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:48:34.705825473 (59h12m44.428388503s) [seg_pos={0 0}, upload=56802]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:48:34.943784077 (59h12m44.666347117s) [upload=56802]: Done uploading segments... Jul 26 22:48:34.943806619 (59h12m44.666369659s) [upload=56802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.239861343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.........................X..........X.X........X.X......] duration: 5.707683449s FAIL (2025-07-26 22:48:34 UTC) orbiter warning: uplink: too many requests Duration: 5.707708206s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.929161232s FAIL (2025-07-26 22:48:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.929186439s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [............................X...X...............X..XX...X.X.] duration: 14.742852662s FAIL (2025-07-26 22:48:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.742863723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX..X.X...X..........................................X.....] duration: 6.844603106s FAIL (2025-07-26 22:48:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.844635498s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.............................X...X...............X..XX...X.X] duration: 1m0.000765405s FAIL (2025-07-26 22:47:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000805883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.........................X..........X.X........X.X.....] duration: 5.964915138s FAIL (2025-07-26 22:47:46 UTC) orbiter warning: uplink: too many requests Duration: 5.964939144s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.131664817s FAIL (2025-07-26 22:47:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.131686189s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXX] duration: 1m0.232567039s FAIL (2025-07-26 22:47:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 14 but got 13) Jul 26 22:46:34.871825749 (59h10m44.594388789s) [upload=56789]: Starting upload Jul 26 22:46:34.871888158 (59h10m44.594451198s) [upload=56789]: Done starting upload Jul 26 22:46:34.871990144 (59h10m44.594553184s) [upload=56789]: Uploading segments... Jul 26 22:46:34.872084946 (59h10m44.594647986s) [upload=56789]: Got next segment. Inline: false Jul 26 22:46:34.872111497 (59h10m44.594674537s) [upload=56789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:46:34.872116135 (59h10m44.594679165s) [upload=56789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:46:35.318303330 (59h10m45.04086637s) [seg_pos={0 0}, upload=56789]: Begin upload segment... Jul 26 22:46:35.318352805 (59h10m45.040915845s) [seg_pos={0 0}, upload=56789]: Done begin upload segment. Jul 26 22:46:35.318374326 (59h10m45.040937366s) [noise=true , address=79.127.219.38:4013 , piece_id=SXXT2ABKYJLIFE52GAM3TVU3L4LDXEXLADT3EDEJKIVBWIMTBRIA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.318410515 (59h10m45.040973555s) [noise=true , address=64.49.13.170:4721 , piece_id=HNTGM5RLDBXMJIEQ4RWH74XYQAS3CNZRKP7BSR6JBRLG4CLHUCCQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.318714326 (59h10m45.041277366s) [noise=true , address=79.127.205.235:4681 , piece_id=VMBI4PVHX6OBXGXUM7ADPQCYMZ5H3PT5LKKGO5FWO3SMIPC3J37Q, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.318806362 (59h10m45.041369402s) [noise=true , address=79.127.219.38:4007 , piece_id=WM3UCRZYXTNFQ47UF2FGPSMHSPAXZJN6LTRLICG4WLFAMI7JWIWQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.319096808 (59h10m45.041659848s) [noise=true , address=156.146.43.225:4448 , piece_id=ICPICX2MPKTRT67LARP6DXCGSCF22OMO6IT6J52TU3ZBTN4ZQPPQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.319363458 (59h10m45.041926498s) [noise=true , address=45.91.116.9:4269 , piece_id=DGA4B35VSNXSP2EPGYQ5LB5XXQCZDAEZWOHCQQWJEOLWI33TAACA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.319606052 (59h10m45.042169092s) [noise=true , address=79.127.201.215:4349 , piece_id=B7GEJQRZC2RWVOID35NMFSWXVQTOKIIWX3QE46APDCKHO3VQXG3A, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.319858635 (59h10m45.042421675s) [noise=true , address=77.90.31.19:4846 , piece_id=TZYCZNRRV3Q2JMB55IHYYZU55OKBNJA5C37WAWDZTZGYMZW3OZAA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.320049941 (59h10m45.042612971s) [noise=true , address=45.91.116.9:4267 , piece_id=3LKBVMTQH6FUHLTSBGWDOENETZPYGWWP4UA33X44CRH6NPKLYJZQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.320262057 (59h10m45.042825097s) [noise=true , address=79.127.205.235:4678 , piece_id=AG33SDWBUPCIXH4BVUMOS3ZFROOHNDVADH6RL3D55CEHYAIKOAQQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.320495965 (59h10m45.043059005s) [noise=true , address=79.127.205.245:4248 , piece_id=IPPXLSXAWY6TVBZ4UPFFTTMTH24WYINIAL5UKGYRTPB274BRG7OA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.320693463 (59h10m45.043256503s) [noise=true , address=79.127.219.51:4212 , piece_id=POWBF2V3KWY7DCHHIPGFBTINMXYTEH5JYLDJSXDVXTF23JZ6ANGQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.320886903 (59h10m45.043449943s) [noise=true , address=77.90.31.16:4748 , piece_id=AU7F47BSEDZQ2VHJHG676EVUQHZREJEWVOUXJ34OKJVNULK5G6NQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.321070535 (59h10m45.043633575s) [noise=true , address=79.127.205.245:4251 , piece_id=G22BO6M2ZHUBIEKIJBGSTVK5TYQ75NOPCQPRG2WLCKCSHXYS33EQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.321263945 (59h10m45.043826985s) [noise=true , address=45.91.116.14:4206 , piece_id=4NN7R22AKYWLCHLSOAEPOPFALGK2BWDIF7GGOS77CLVFRM4IUTUA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.321457856 (59h10m45.044020896s) [noise=true , address=79.127.219.47:4061 , piece_id=6XMSMO5ZIKLOLMSOB5WML7Z2PS4WZA5S7LVDRPSKH4A6JJHCLDWQ, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.321642799 (59h10m45.044205839s) [noise=true , address=64.49.13.137:4500 , piece_id=KB5JMPE2YJ3EH4DK3JZIBUS323SBNUA45HCZMEGEOGTVWXCINYKA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:35.321914289 (59h10m45.044477339s) [noise=true , address=79.127.205.246:4279 , piece_id=64IFKV4MS6YIRFHZ3BRGWRKW34HZDXAOHHF7E66PP4C7MY5TMPVA, seg_pos={0 0}, upload=56789]: Uploading piece... Jul 26 22:46:36.384903755 (59h10m46.107466795s) [noise=true , address=79.127.219.47:4061 , piece_id=6XMSMO5ZIKLOLMSOB5WML7Z2PS4WZA5S7LVDRPSKH4A6JJHCLDWQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:36.413372864 (59h10m46.135935904s) [upload=56789]: Next returned nil segment Jul 26 22:46:36.413387453 (59h10m46.135950493s) [upload=56789]: Waiting for error group managing segments... Jul 26 22:46:36.501992019 (59h10m46.224555059s) [noise=true , address=79.127.219.51:4212 , piece_id=POWBF2V3KWY7DCHHIPGFBTINMXYTEH5JYLDJSXDVXTF23JZ6ANGQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:36.695705409 (59h10m46.418268449s) [noise=true , address=79.127.205.246:4279 , piece_id=64IFKV4MS6YIRFHZ3BRGWRKW34HZDXAOHHF7E66PP4C7MY5TMPVA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:36.724593720 (59h10m46.447156771s) [noise=true , address=79.127.205.235:4681 , piece_id=VMBI4PVHX6OBXGXUM7ADPQCYMZ5H3PT5LKKGO5FWO3SMIPC3J37Q, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:36.725373722 (59h10m46.447936763s) [noise=true , address=79.127.205.245:4251 , piece_id=G22BO6M2ZHUBIEKIJBGSTVK5TYQ75NOPCQPRG2WLCKCSHXYS33EQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:36.727511503 (59h10m46.450074543s) [noise=true , address=79.127.205.245:4248 , piece_id=IPPXLSXAWY6TVBZ4UPFFTTMTH24WYINIAL5UKGYRTPB274BRG7OA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:36.742173856 (59h10m46.464736886s) [noise=true , address=79.127.205.235:4678 , piece_id=AG33SDWBUPCIXH4BVUMOS3ZFROOHNDVADH6RL3D55CEHYAIKOAQQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:37.303188038 (59h10m47.025751078s) [noise=true , address=79.127.219.38:4007 , piece_id=WM3UCRZYXTNFQ47UF2FGPSMHSPAXZJN6LTRLICG4WLFAMI7JWIWQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:37.623065718 (59h10m47.345628788s) [noise=true , address=79.127.219.38:4013 , piece_id=SXXT2ABKYJLIFE52GAM3TVU3L4LDXEXLADT3EDEJKIVBWIMTBRIA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:37.631461651 (59h10m47.354024691s) [noise=true , address=64.49.13.170:4721 , piece_id=HNTGM5RLDBXMJIEQ4RWH74XYQAS3CNZRKP7BSR6JBRLG4CLHUCCQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:37.728177887 (59h10m47.450740927s) [noise=true , address=64.49.13.137:4500 , piece_id=KB5JMPE2YJ3EH4DK3JZIBUS323SBNUA45HCZMEGEOGTVWXCINYKA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:39.175459395 (59h10m48.898022435s) [noise=true , address=156.146.43.225:4448 , piece_id=ICPICX2MPKTRT67LARP6DXCGSCF22OMO6IT6J52TU3ZBTN4ZQPPQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:39.225002105 (59h10m48.947565145s) [noise=true , address=45.91.116.9:4267 , piece_id=3LKBVMTQH6FUHLTSBGWDOENETZPYGWWP4UA33X44CRH6NPKLYJZQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:40.068505201 (59h10m49.791068241s) [noise=true , address=45.91.116.14:4206 , piece_id=4NN7R22AKYWLCHLSOAEPOPFALGK2BWDIF7GGOS77CLVFRM4IUTUA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:46:40.322843303 (59h10m50.045406333s) [noise=true , address=77.90.31.19:4846 , piece_id=TZYCZNRRV3Q2JMB55IHYYZU55OKBNJA5C37WAWDZTZGYMZW3OZAA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:44.754977178 (59h10m54.477540218s) [noise=true , address=79.127.201.215:4349 , piece_id=B7GEJQRZC2RWVOID35NMFSWXVQTOKIIWX3QE46APDCKHO3VQXG3A, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:46:48.494153573 (59h10m58.216716653s) [noise=true , address=45.91.116.9:4269 , piece_id=DGA4B35VSNXSP2EPGYQ5LB5XXQCZDAEZWOHCQQWJEOLWI33TAACA, seg_pos={0 0}, upload=56789]: Done uploading piece. err: <nil> Jul 26 22:47:34.225626765 (59h11m43.948189806s) [noise=true , address=77.90.31.16:4748 , piece_id=AU7F47BSEDZQ2VHJHG676EVUQHZREJEWVOUXJ34OKJVNULK5G6NQ, seg_pos={0 0}, upload=56789]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; context deadline exceeded Jul 26 22:47:34.225662524 (59h11m43.948225554s) [seg_pos={0 0}, upload=56789]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:47:34.456391114 (59h11m44.178954164s) [upload=56789]: Done uploading segments... Jul 26 22:47:34.456423576 (59h11m44.178986706s) [upload=56789]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.232580435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXX..X.X...X..........................................X....] duration: 5.71786523s FAIL (2025-07-26 22:47:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.717891571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.........................X..........X.X........X.X....] duration: 5.762769556s FAIL (2025-07-26 22:46:53 UTC) orbiter warning: uplink: too many requests Duration: 5.762794503s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.960742345s FAIL (2025-07-26 22:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.960764256s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XXXX..X.X...X..........................................X...] duration: 5.780127766s FAIL (2025-07-26 22:46:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780159347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXX] duration: 1m0.233056545s FAIL (2025-07-26 22:46:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 5:33.523948130 (59h9m43.24651117s) [upload=56756]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:45:34.408142871 (59h9m44.130705911s) [upload=56772]: Starting upload Jul 26 22:45:34.408197786 (59h9m44.130760826s) [upload=56772]: Done starting upload Jul 26 22:45:34.408312796 (59h9m44.130875836s) [upload=56772]: Uploading segments... Jul 26 22:45:34.408422706 (59h9m44.130985756s) [upload=56772]: Got next segment. Inline: false Jul 26 22:45:34.408455970 (59h9m44.13101901s) [upload=56772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:45:34.408499312 (59h9m44.131062352s) [upload=56772]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:45:34.852916551 (59h9m44.575479591s) [seg_pos={0 0}, upload=56772]: Begin upload segment... Jul 26 22:45:34.852957099 (59h9m44.575520129s) [seg_pos={0 0}, upload=56772]: Done begin upload segment. Jul 26 22:45:34.852964543 (59h9m44.575527583s) [noise=true , address=79.127.205.235:4690 , piece_id=EJSJN7JA4TFNDDPJGXZK4YWTMZAOQQI6QFWMKNKOPXIJJETZNYZQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.852974983 (59h9m44.575538023s) [noise=true , address=156.146.43.225:4434 , piece_id=XEL6DFFJBZAJJ2F5T4L7AWMYEB3ORMMU5EF7JNKNR4GEO2WFUHTQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.853304322 (59h9m44.575867362s) [noise=true , address=45.91.116.11:4121 , piece_id=IXWNGMIAPAPDBK67JVHDSLW2E3IGKTOYJA2YZSSOYNX6OKDYDITQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.853366141 (59h9m44.575929181s) [noise=true , address=79.127.219.51:4212 , piece_id=4UMUAJC4RTFIJKQLQ2UVXLCOB3Y5UVEMJUL2HPAAUEUMRPAA4ANA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.853553861 (59h9m44.576116891s) [noise=true , address=79.127.219.48:4090 , piece_id=GGO7NTT4ZQBE6RMSCJZWVUVWIYTC42BX2UHYUJ6OQ4BKX5GUXRYQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.853790333 (59h9m44.576353373s) [noise=true , address=79.127.201.215:4344 , piece_id=YURMTHIUSSXU5PJCUH73NUS73JUBKM2H3Y2OKW6EOARNWHPD42EA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.854051493 (59h9m44.576614533s) [noise=true , address=77.90.31.16:4760 , piece_id=HVHFXUJLWNGSLBMYAEPPGSNYCVK2YWCOTNVY6DUIIPYZIFKCRM3Q, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.854300239 (59h9m44.576863309s) [noise=true , address=64.49.13.170:4719 , piece_id=K3B4SXSWCZ5EU6YWDGEYEEBHSOST3ZNOV6NB7QSKWPHGL2UROSWQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.854450696 (59h9m44.577013786s) [noise=true , address=104.36.82.19:4425 , piece_id=Y34IZRDQYVCSQ3XHTZLMCXIZHCEBYI3GVJNW5QPSCWDKBA3ZCBKA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.854768094 (59h9m44.577331124s) [noise=true , address=77.90.31.21:4891 , piece_id=V5RZ5WFCL5MNF2P2YJGZRSW2AP3DQ2FMVCQZC7HFI727KMDJD2HA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855065583 (59h9m44.577628623s) [noise=true , address=79.127.205.245:4227 , piece_id=BLSHZXDATOSLL2JV4AN7O6Q4GTS7N4FRXNYIAOTC4NEWBNRTQUJA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855202083 (59h9m44.577765123s) [noise=true , address=77.90.31.18:4813 , piece_id=I55MGKIFLKXDWRTH55H4Z23MOMLBH3T3LYOYBP7MC4CAAAU3ISFA, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855410753 (59h9m44.577973793s) [noise=true , address=79.127.205.234:4619 , piece_id=XPGX25D2SXKJAQ3E5ANSWC4ZQXZB6MTGU4RTM2ZZGFZJMSGI7DUQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855606928 (59h9m44.578169968s) [noise=true , address=77.90.31.12:4605 , piece_id=EE47UH477S7DMLAHJ3CYC3ZUEH3U7F6ZMBIZ657ORU2UPWWTXA2Q, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855672975 (59h9m44.578236015s) [noise=true , address=45.91.116.9:4288 , piece_id=2CSVHGD33J3MLSTRIBQXMXVS5S5QV4VN5CNGKARGKWSVDPDZBDHQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.855928444 (59h9m44.578491484s) [noise=true , address=45.91.116.14:4206 , piece_id=N345VKV2XXS4QHRJO4ZHC5XSGWBI22EF7ZZHST6FVUPT64MZVPAQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.856147252 (59h9m44.578710292s) [noise=true , address=79.127.219.47:4061 , piece_id=VMTH2ZA5AJ57QHGCU62WLZIVQT6PXGVMHJZR5XHRC3FN7WNOUNWQ, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:34.856234339 (59h9m44.578797379s) [noise=true , address=45.91.116.8:4065 , piece_id=ESWXCUEMR2QO73JS2L4V6HSQD7QQB7FLHPVQKWUOQNDH3OOSIN6A, seg_pos={0 0}, upload=56772]: Uploading piece... Jul 26 22:45:35.885642947 (59h9m45.608205987s) [noise=true , address=156.146.43.225:4434 , piece_id=XEL6DFFJBZAJJ2F5T4L7AWMYEB3ORMMU5EF7JNKNR4GEO2WFUHTQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:35.939197803 (59h9m45.661760843s) [upload=56772]: Next returned nil segment Jul 26 22:45:35.939215748 (59h9m45.661778788s) [upload=56772]: Waiting for error group managing segments... Jul 26 22:45:36.030913171 (59h9m45.753476221s) [noise=true , address=79.127.219.51:4212 , piece_id=4UMUAJC4RTFIJKQLQ2UVXLCOB3Y5UVEMJUL2HPAAUEUMRPAA4ANA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:36.217475729 (59h9m45.940038769s) [noise=true , address=79.127.205.245:4227 , piece_id=BLSHZXDATOSLL2JV4AN7O6Q4GTS7N4FRXNYIAOTC4NEWBNRTQUJA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:36.239026321 (59h9m45.961589351s) [noise=true , address=79.127.205.234:4619 , piece_id=XPGX25D2SXKJAQ3E5ANSWC4ZQXZB6MTGU4RTM2ZZGFZJMSGI7DUQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:36.285719169 (59h9m46.008282199s) [noise=true , address=79.127.205.235:4690 , piece_id=EJSJN7JA4TFNDDPJGXZK4YWTMZAOQQI6QFWMKNKOPXIJJETZNYZQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:36.798286538 (59h9m46.520849568s) [noise=true , address=77.90.31.21:4891 , piece_id=V5RZ5WFCL5MNF2P2YJGZRSW2AP3DQ2FMVCQZC7HFI727KMDJD2HA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:36.876640702 (59h9m46.599203752s) [noise=true , address=104.36.82.19:4425 , piece_id=Y34IZRDQYVCSQ3XHTZLMCXIZHCEBYI3GVJNW5QPSCWDKBA3ZCBKA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:38.421659609 (59h9m48.144222649s) [noise=true , address=79.127.219.47:4061 , piece_id=VMTH2ZA5AJ57QHGCU62WLZIVQT6PXGVMHJZR5XHRC3FN7WNOUNWQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:39.327460468 (59h9m49.050023508s) [noise=true , address=79.127.219.48:4090 , piece_id=GGO7NTT4ZQBE6RMSCJZWVUVWIYTC42BX2UHYUJ6OQ4BKX5GUXRYQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:39.765198467 (59h9m49.487761497s) [noise=true , address=45.91.116.8:4065 , piece_id=ESWXCUEMR2QO73JS2L4V6HSQD7QQB7FLHPVQKWUOQNDH3OOSIN6A, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:41.488916549 (59h9m51.211479599s) [noise=true , address=64.49.13.170:4719 , piece_id=K3B4SXSWCZ5EU6YWDGEYEEBHSOST3ZNOV6NB7QSKWPHGL2UROSWQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:41.525182857 (59h9m51.247745887s) [noise=true , address=45.91.116.11:4121 , piece_id=IXWNGMIAPAPDBK67JVHDSLW2E3IGKTOYJA2YZSSOYNX6OKDYDITQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 22:45:42.458358642 (59h9m52.180921682s) [noise=true , address=45.91.116.9:4288 , piece_id=2CSVHGD33J3MLSTRIBQXMXVS5S5QV4VN5CNGKARGKWSVDPDZBDHQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:47.606153132 (59h9m57.328716172s) [noise=true , address=45.91.116.14:4206 , piece_id=N345VKV2XXS4QHRJO4ZHC5XSGWBI22EF7ZZHST6FVUPT64MZVPAQ, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:45:47.906192816 (59h9m57.628755856s) [noise=true , address=77.90.31.18:4813 , piece_id=I55MGKIFLKXDWRTH55H4Z23MOMLBH3T3LYOYBP7MC4CAAAU3ISFA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:46:02.891764489 (59h10m12.614327529s) [noise=true , address=79.127.201.215:4344 , piece_id=YURMTHIUSSXU5PJCUH73NUS73JUBKM2H3Y2OKW6EOARNWHPD42EA, seg_pos={0 0}, upload=56772]: Done uploading piece. err: <nil> Jul 26 22:46:33.762692507 (59h10m43.485255557s) [noise=true , address=77.90.31.16:4760 , piece_id=HVHFXUJLWNGSLBMYAEPPGSNYCVK2YWCOTNVY6DUIIPYZIFKCRM3Q, seg_pos={0 0}, upload=56772]: Done uploading piece. err: ecclient: upload failed (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, address:77.90.31.16:4760): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:46:33.762717103 (59h10m43.485280154s) [noise=true , address=77.90.31.12:4605 , piece_id=EE47UH477S7DMLAHJ3CYC3ZUEH3U7F6ZMBIZ657ORU2UPWWTXA2Q, seg_pos={0 0}, upload=56772]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: write tcp 188.166.225.170:52426->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:46:33.762756650 (59h10m43.48531969s) [seg_pos={0 0}, upload=56772]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:46:33.994007429 (59h10m43.71657047s) [upload=56772]: Done uploading segments... Jul 26 22:46:33.994032827 (59h10m43.716595867s) [upload=56772]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.233067435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............................X...X...............X..XX...X.] duration: 10.770752766s FAIL (2025-07-26 22:46:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.770768937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.........................X..........X.X........X.X...] duration: 5.664073672s FAIL (2025-07-26 22:46:15 UTC) orbiter warning: uplink: too many requests Duration: 5.664107706s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...............................X...X...............X..XX...X] duration: 1m0.001253848s FAIL (2025-07-26 22:46:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001278456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.158665027s FAIL (2025-07-26 22:45:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.158685637s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XXXX..X.X...X..........................................X..] duration: 6.015077462s FAIL (2025-07-26 22:45:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.015106699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.....X.........................X..........X.X........X.X..] duration: 5.807132271s FAIL (2025-07-26 22:45:41 UTC) orbiter warning: uplink: too many requests Duration: 5.807159545s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXX] duration: 1m0.234390216s FAIL (2025-07-26 22:45:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nough pieces (needed at least 14 but got 13) Jul 26 22:44:33.946711819 (59h8m43.66927489s) [upload=56756]: Starting upload Jul 26 22:44:33.946831869 (59h8m43.669394909s) [upload=56756]: Done starting upload Jul 26 22:44:33.946955966 (59h8m43.669519046s) [upload=56756]: Uploading segments... Jul 26 22:44:33.947256231 (59h8m43.669819301s) [upload=56756]: Got next segment. Inline: false Jul 26 22:44:33.947309583 (59h8m43.669872663s) [upload=56756]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:44:33.947317940 (59h8m43.66988101s) [upload=56756]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:44:34.399197706 (59h8m44.121760746s) [seg_pos={0 0}, upload=56756]: Begin upload segment... Jul 26 22:44:34.399252811 (59h8m44.121815852s) [seg_pos={0 0}, upload=56756]: Done begin upload segment. Jul 26 22:44:34.399275445 (59h8m44.121838485s) [noise=true , address=79.127.219.38:4009 , piece_id=PGBEIRCCYD4DWTI7WKW6PPTT2BEYZXFJK6DWH5LA6C3VXPLHIHWQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.399320290 (59h8m44.12188333s) [noise=true , address=104.36.82.7:4936 , piece_id=FYT5ZYPSSJNGIYJPSULECPZPE5ULVTBV622AAPF7I7TRSXMIASIA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.399706038 (59h8m44.122269078s) [noise=true , address=77.90.31.19:4860 , piece_id=JBSTR5PICM3VVJ7TONOI56HWKY246USCX7N2GDBSBNPOIERF23HA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.399784729 (59h8m44.122347769s) [noise=true , address=79.127.205.233:4642 , piece_id=RMNFMBD55SRHVRB7V3COW7WLCJ4D4N2HTUZD4F7CLV3WN4YRY2JQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.399976897 (59h8m44.122539937s) [noise=true , address=193.149.164.143:4696 , piece_id=T5IHB6TKSDLEEDO47JVEICTS3EIVW4B5TPWW7BDGPQAIAXFG4OGA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.400251782 (59h8m44.122814823s) [noise=true , address=79.127.219.48:4090 , piece_id=6PQIZD4BFRHBVVZCSDE7TI2CVQUWW7VAZ4W6MZJANUM7B66YZUIQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.400543371 (59h8m44.123106411s) [noise=true , address=64.49.13.138:4482 , piece_id=JFXMSHI3SDJIZDZJVA2ZDVJZUXAHQVC7A4EKFB7YDELYRBFMRRVA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.400688057 (59h8m44.123251097s) [noise=true , address=77.90.31.17:4800 , piece_id=JTT745BTM7KCE6V3YQ5TTDMWNYHO5E7DMGU4SKZT4L3A2RJURXEQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.401037256 (59h8m44.123600286s) [noise=true , address=64.49.13.170:4719 , piece_id=LOE2PXQ6SPNMWHLRPTV3RDRTN5W43LHHROW62PWM2HS4GIKMIRVQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.401324294 (59h8m44.123887325s) [noise=true , address=77.90.31.10:4531 , piece_id=AHBBG63W54OGP4W6ENZQT2CIBZ7BQZQOVSXZVZS4GEBR3XCUH3PQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.401554925 (59h8m44.124117955s) [noise=true , address=79.127.220.97:4361 , piece_id=VNOJYTJZYMJHZSHCBB363O44U6UA6ZJHXKH3HRPF2LGIMNSZDV7A, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.401832527 (59h8m44.124395567s) [noise=true , address=79.127.205.246:4282 , piece_id=J6DJQDM2RKXRKNBEK64XDK35ZFTF2BNYAHOGBOEESDLJHSZTCWVA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.402661834 (59h8m44.125224874s) [noise=true , address=79.127.201.215:4341 , piece_id=OW37NVBZFL4FEPSE3GKK35L53T5JKL6CPCNLBAAFGPU4OMOYBOOQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.402956347 (59h8m44.125519387s) [noise=true , address=77.90.31.12:4617 , piece_id=XNLB3YGLSM47PYUSHGTV6DPWDY6ANKYL4EM7TDBMN2K5Y6FRRI7Q, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.403196747 (59h8m44.125759787s) [noise=true , address=156.146.43.225:4448 , piece_id=4YCT5ATBDKIVMUFJCTEG3WQBXOIAV4X33BP6MERD5BAAI6WUU7UA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.404187412 (59h8m44.126750452s) [noise=true , address=77.90.31.18:4816 , piece_id=43ZMLISDXWGFE3NPCYQC3A65XTFVVAAKGJW7YTJURNHI2ATYYIAQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.404501534 (59h8m44.127064564s) [noise=true , address=104.36.82.22:4527 , piece_id=6KVRZYOJW4AQLNN4BHP426SZ32HUPXXBMDGHWMQ7SHA5UP6ITJDA, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:34.405260165 (59h8m44.127823195s) [noise=true , address=79.127.205.245:4251 , piece_id=HYPLKDN6BE2Y52SJDO6EGS63DT6LSZ4J2TDTVOBS6QJSDJHIQDSQ, seg_pos={0 0}, upload=56756]: Uploading piece... Jul 26 22:44:35.523455856 (59h8m45.246018896s) [upload=56756]: Next returned nil segment Jul 26 22:44:35.523474693 (59h8m45.246037733s) [upload=56756]: Waiting for error group managing segments... Jul 26 22:44:35.709061193 (59h8m45.431624233s) [noise=true , address=64.49.13.170:4719 , piece_id=LOE2PXQ6SPNMWHLRPTV3RDRTN5W43LHHROW62PWM2HS4GIKMIRVQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: ecclient: upload failed (node:126nRFki7uLyG9xwY8t6BeAumn8ME2HwccdC2EAwsiEV1p2juZ9, address:64.49.13.170:4719): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:35.808374709 (59h8m45.530937739s) [noise=true , address=79.127.205.246:4282 , piece_id=J6DJQDM2RKXRKNBEK64XDK35ZFTF2BNYAHOGBOEESDLJHSZTCWVA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:35.842772051 (59h8m45.565335091s) [noise=true , address=79.127.205.245:4251 , piece_id=HYPLKDN6BE2Y52SJDO6EGS63DT6LSZ4J2TDTVOBS6QJSDJHIQDSQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:35.859134987 (59h8m45.581698027s) [noise=true , address=79.127.205.233:4642 , piece_id=RMNFMBD55SRHVRB7V3COW7WLCJ4D4N2HTUZD4F7CLV3WN4YRY2JQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:36.316788020 (59h8m46.03935106s) [noise=true , address=79.127.219.38:4009 , piece_id=PGBEIRCCYD4DWTI7WKW6PPTT2BEYZXFJK6DWH5LA6C3VXPLHIHWQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:36.749595607 (59h8m46.472158647s) [noise=true , address=79.127.219.48:4090 , piece_id=6PQIZD4BFRHBVVZCSDE7TI2CVQUWW7VAZ4W6MZJANUM7B66YZUIQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:36.910295833 (59h8m46.632858873s) [noise=true , address=79.127.220.97:4361 , piece_id=VNOJYTJZYMJHZSHCBB363O44U6UA6ZJHXKH3HRPF2LGIMNSZDV7A, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:37.320615423 (59h8m47.043178463s) [noise=true , address=64.49.13.138:4482 , piece_id=JFXMSHI3SDJIZDZJVA2ZDVJZUXAHQVC7A4EKFB7YDELYRBFMRRVA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:44:39.023560199 (59h8m48.746123229s) [noise=true , address=156.146.43.225:4448 , piece_id=4YCT5ATBDKIVMUFJCTEG3WQBXOIAV4X33BP6MERD5BAAI6WUU7UA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:39.752722998 (59h8m49.475286038s) [noise=true , address=79.127.201.215:4341 , piece_id=OW37NVBZFL4FEPSE3GKK35L53T5JKL6CPCNLBAAFGPU4OMOYBOOQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:40.392232775 (59h8m50.114795815s) [noise=true , address=193.149.164.143:4696 , piece_id=T5IHB6TKSDLEEDO47JVEICTS3EIVW4B5TPWW7BDGPQAIAXFG4OGA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:44.707589632 (59h8m54.430152662s) [noise=true , address=77.90.31.18:4816 , piece_id=43ZMLISDXWGFE3NPCYQC3A65XTFVVAAKGJW7YTJURNHI2ATYYIAQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:48.629785748 (59h8m58.352348778s) [noise=true , address=77.90.31.19:4860 , piece_id=JBSTR5PICM3VVJ7TONOI56HWKY246USCX7N2GDBSBNPOIERF23HA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:44:48.904336573 (59h8m58.626899613s) [noise=true , address=104.36.82.22:4527 , piece_id=6KVRZYOJW4AQLNN4BHP426SZ32HUPXXBMDGHWMQ7SHA5UP6ITJDA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:45:08.598936932 (59h9m18.321499972s) [noise=true , address=104.36.82.7:4936 , piece_id=FYT5ZYPSSJNGIYJPSULECPZPE5ULVTBV622AAPF7I7TRSXMIASIA, seg_pos={0 0}, upload=56756]: Done uploading piece. err: <nil> Jul 26 22:45:33.291983054 (59h9m43.014546104s) [noise=true , address=77.90.31.12:4617 , piece_id=XNLB3YGLSM47PYUSHGTV6DPWDY6ANKYL4EM7TDBMN2K5Y6FRRI7Q, seg_pos={0 0}, upload=56756]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: write tcp 188.166.225.170:58782->77.90.31.12:4617: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:45:33.292167998 (59h9m43.014731038s) [noise=true , address=77.90.31.17:4800 , piece_id=JTT745BTM7KCE6V3YQ5TTDMWNYHO5E7DMGU4SKZT4L3A2RJURXEQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: ecclient: upload failed (node:1TxoPpye17ET1GR52xy2EmkUdq83x4T9TsfuwrKr8cYXf3HLYt, address:77.90.31.17:4800): protocol: read tcp 188.166.225.170:36882->77.90.31.17:4800: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:45:33.292282737 (59h9m43.014845767s) [noise=true , address=77.90.31.10:4531 , piece_id=AHBBG63W54OGP4W6ENZQT2CIBZ7BQZQOVSXZVZS4GEBR3XCUH3PQ, seg_pos={0 0}, upload=56756]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:45:33.292296474 (59h9m43.014859504s) [seg_pos={0 0}, upload=56756]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:45:33.523909426 (59h9m43.246472466s) [upload=56756]: Done uploading segments... Jul 26 22:45:33.523948130 (59h9m43.24651117s) [upload=56756]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.23440287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................................X...X...............X..XX...] duration: 9.629853405s FAIL (2025-07-26 22:44:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.629877782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..XXXX..X.X...X..........................................X.] duration: 6.03251102s FAIL (2025-07-26 22:44:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.032560183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.123575562s FAIL (2025-07-26 22:44:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.123592263s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.....X.........................X..........X.X........X.X.] duration: 5.70211159s FAIL (2025-07-26 22:44:41 UTC) orbiter warning: uplink: too many requests Duration: 5.702146997s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XX] duration: 1m0.23540599s FAIL (2025-07-26 22:44:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : <nil> Jul 26 22:43:27.064504095 (59h7m36.787067135s) [upload=56728]: Done uploading segments... Jul 26 22:43:27.064528242 (59h7m36.787091282s) [upload=56728]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Jul 26 22:43:33.470328013 (59h7m43.192891053s) [upload=56743]: Starting upload Jul 26 22:43:33.470390051 (59h7m43.192953091s) [upload=56743]: Done starting upload Jul 26 22:43:33.470532794 (59h7m43.193095834s) [upload=56743]: Uploading segments... Jul 26 22:43:33.470627386 (59h7m43.193190426s) [upload=56743]: Got next segment. Inline: false Jul 26 22:43:33.470657273 (59h7m43.193220313s) [upload=56743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:33.470663024 (59h7m43.193226054s) [upload=56743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:33.975842378 (59h7m43.698405418s) [seg_pos={0 0}, upload=56743]: Begin upload segment... Jul 26 22:43:33.975894007 (59h7m43.698457047s) [seg_pos={0 0}, upload=56743]: Done begin upload segment. Jul 26 22:43:33.975936078 (59h7m43.698499118s) [noise=true , address=64.49.13.155:4918 , piece_id=MQE3AUCUAQDLIBDG7BZGJPXPKSJ6IUBOAECK4HXROZCUQQ7NX7NQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.975991093 (59h7m43.698554133s) [noise=true , address=45.91.116.9:4269 , piece_id=776PUCLI2O7GBMP5HZ4DRVR3AVJRHUZA2D5QCXES6UUTUCPHMPEQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.976446323 (59h7m43.699009363s) [noise=true , address=79.127.220.101:4048 , piece_id=EDRARNUQOGQFLGSKJQJ6SLXVAPIVFFYVPRD7GBUPFQ4WJXQRPG7A, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.976474868 (59h7m43.699037908s) [noise=true , address=79.127.219.38:4017 , piece_id=PYHZYYCHB3T2WTR2ZG67L2QYPJDVK4ODXQ3VYOMH6ID7K4O6MYOQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.976734726 (59h7m43.699297766s) [noise=true , address=156.146.43.225:4448 , piece_id=GE3B6LD4DGRJFWOYGAWXO33BKTXWQ3KUH5N2AS2YHXHXL4VGXLGQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.977002427 (59h7m43.699565467s) [noise=true , address=45.91.116.13:4175 , piece_id=UCZ52LTCGNLZO62EF62YXEL7N7DSL4DRTZ7TLSVVG3INK47QX7VQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.977254741 (59h7m43.699817771s) [noise=true , address=77.90.31.19:4875 , piece_id=67W6PRVOCJRJI3OGVNN7WPHYRYQ2TUBL46HOSTBXMS2DY24ISFKQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.977511922 (59h7m43.700074962s) [noise=true , address=77.90.31.18:4815 , piece_id=DNBQVIVMTVBD7Q2T3M2CXFSK4S6D76JXO6CAPSFHFEUGBEJLT5KA, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.977756711 (59h7m43.700319751s) [noise=true , address=79.127.205.235:4691 , piece_id=WHAYGKWW7XFPPYU2BSC2D5TX5IK7GHA5GQ2GBWZ5NGZR23V5543Q, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.978023251 (59h7m43.700586281s) [noise=true , address=79.127.205.245:4248 , piece_id=RA5I2M4KPXRDDDBGDCYID65WOMA3XMAZWWWCNR5RJMPZSAKBSV2A, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.978215107 (59h7m43.700778137s) [noise=true , address=64.49.13.140:4514 , piece_id=P4PKSOSVNXURMKD6QDRQE6KE3YNT4CJAFWGOS7NKWFJ6N2DOG3ZQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.978413848 (59h7m43.700976878s) [noise=true , address=79.127.219.51:4212 , piece_id=LWML7EEGCIPVT6FDCVCNAU7C2WBESHYVX3CPXJ7HI5DCIQQQ4Q6A, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.978611416 (59h7m43.701174456s) [noise=true , address=79.127.219.47:4061 , piece_id=H2NL7BJDLP6ARCW4YE2ETMC7HAGGW3VVHUNMLPW3THMHKAJD4ZMQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.978811789 (59h7m43.701374829s) [noise=true , address=77.90.31.10:4539 , piece_id=6RNFNGE5NM7XCLJJECZIHUVYP6AMN7KWSJK5ZMIM26YTD6KBTAXA, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.979019928 (59h7m43.701582968s) [noise=true , address=77.90.31.15:4730 , piece_id=WCFLFFLSMLOOUMVVQN6AZFUZDES7C7UYNRM2PJQE6MDXJ263FDDQ, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.979240830 (59h7m43.70180386s) [noise=true , address=79.127.205.233:4635 , piece_id=5PKOMJNPFK5J77IJE43TACQCRV5U3NZ6HNZ2LM3UPLYTSCRA3RRA, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.979437767 (59h7m43.702000807s) [noise=true , address=79.127.205.234:4589 , piece_id=TFZGZYR6WGAIKKEKHEYOOGHSSZMHOUK6NZ2KBN3KGRCA2LUKCHOA, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:33.979684199 (59h7m43.702247229s) [noise=true , address=45.91.116.5:4310 , piece_id=BIKPCN526SKKENA6JTY2LOVEIXOULEFH7FJAW5W65QOJ6CXJ7VCA, seg_pos={0 0}, upload=56743]: Uploading piece... Jul 26 22:43:34.997379344 (59h7m44.719942374s) [noise=true , address=79.127.219.47:4061 , piece_id=H2NL7BJDLP6ARCW4YE2ETMC7HAGGW3VVHUNMLPW3THMHKAJD4ZMQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:35.047267474 (59h7m44.769830514s) [noise=true , address=79.127.219.51:4212 , piece_id=LWML7EEGCIPVT6FDCVCNAU7C2WBESHYVX3CPXJ7HI5DCIQQQ4Q6A, seg_pos={0 0}, upload=56743]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:35.095442607 (59h7m44.818005637s) [upload=56743]: Next returned nil segment Jul 26 22:43:35.095459259 (59h7m44.818022299s) [upload=56743]: Waiting for error group managing segments... Jul 26 22:43:35.390390968 (59h7m45.112954008s) [noise=true , address=79.127.205.245:4248 , piece_id=RA5I2M4KPXRDDDBGDCYID65WOMA3XMAZWWWCNR5RJMPZSAKBSV2A, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:35.390432738 (59h7m45.112995778s) [noise=true , address=79.127.205.234:4589 , piece_id=TFZGZYR6WGAIKKEKHEYOOGHSSZMHOUK6NZ2KBN3KGRCA2LUKCHOA, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:35.393336053 (59h7m45.115899083s) [noise=true , address=79.127.205.233:4635 , piece_id=5PKOMJNPFK5J77IJE43TACQCRV5U3NZ6HNZ2LM3UPLYTSCRA3RRA, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:35.407296112 (59h7m45.129859152s) [noise=true , address=79.127.205.235:4691 , piece_id=WHAYGKWW7XFPPYU2BSC2D5TX5IK7GHA5GQ2GBWZ5NGZR23V5543Q, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:35.883676275 (59h7m45.606239315s) [noise=true , address=79.127.219.38:4017 , piece_id=PYHZYYCHB3T2WTR2ZG67L2QYPJDVK4ODXQ3VYOMH6ID7K4O6MYOQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:36.202435695 (59h7m45.924998735s) [noise=true , address=77.90.31.10:4539 , piece_id=6RNFNGE5NM7XCLJJECZIHUVYP6AMN7KWSJK5ZMIM26YTD6KBTAXA, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:37.448464988 (59h7m47.171028028s) [noise=true , address=45.91.116.9:4269 , piece_id=776PUCLI2O7GBMP5HZ4DRVR3AVJRHUZA2D5QCXES6UUTUCPHMPEQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:38.881993523 (59h7m48.604556553s) [noise=true , address=156.146.43.225:4448 , piece_id=GE3B6LD4DGRJFWOYGAWXO33BKTXWQ3KUH5N2AS2YHXHXL4VGXLGQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:39.215586772 (59h7m48.938149812s) [noise=true , address=45.91.116.5:4310 , piece_id=BIKPCN526SKKENA6JTY2LOVEIXOULEFH7FJAW5W65QOJ6CXJ7VCA, seg_pos={0 0}, upload=56743]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:39.501225602 (59h7m49.223788642s) [noise=true , address=79.127.220.101:4048 , piece_id=EDRARNUQOGQFLGSKJQJ6SLXVAPIVFFYVPRD7GBUPFQ4WJXQRPG7A, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:39.829088860 (59h7m49.55165189s) [noise=true , address=64.49.13.155:4918 , piece_id=MQE3AUCUAQDLIBDG7BZGJPXPKSJ6IUBOAECK4HXROZCUQQ7NX7NQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: ecclient: upload failed (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK, address:64.49.13.155:4918): protocol: storage node overloaded, request limit: 2000 Jul 26 22:43:43.398908728 (59h7m53.121471768s) [noise=true , address=64.49.13.140:4514 , piece_id=P4PKSOSVNXURMKD6QDRQE6KE3YNT4CJAFWGOS7NKWFJ6N2DOG3ZQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:43:49.789882158 (59h7m59.512445188s) [noise=true , address=77.90.31.19:4875 , piece_id=67W6PRVOCJRJI3OGVNN7WPHYRYQ2TUBL46HOSTBXMS2DY24ISFKQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:44:20.585330902 (59h8m30.307893932s) [noise=true , address=45.91.116.13:4175 , piece_id=UCZ52LTCGNLZO62EF62YXEL7N7DSL4DRTZ7TLSVVG3INK47QX7VQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:44:22.517559765 (59h8m32.240122826s) [noise=true , address=77.90.31.18:4815 , piece_id=DNBQVIVMTVBD7Q2T3M2CXFSK4S6D76JXO6CAPSFHFEUGBEJLT5KA, seg_pos={0 0}, upload=56743]: Done uploading piece. err: <nil> Jul 26 22:44:32.831267152 (59h8m42.553830192s) [noise=true , address=77.90.31.15:4730 , piece_id=WCFLFFLSMLOOUMVVQN6AZFUZDES7C7UYNRM2PJQE6MDXJ263FDDQ, seg_pos={0 0}, upload=56743]: Done uploading piece. err: ecclient: upload failed (node:1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, address:77.90.31.15:4730): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:44:32.831338167 (59h8m42.553901207s) [seg_pos={0 0}, upload=56743]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:44:33.064611417 (59h8m42.787174457s) [upload=56743]: Done uploading segments... Jul 26 22:44:33.064636835 (59h8m42.787199915s) [upload=56743]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.235427302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX.....X.........................X..........X.X........X.X] duration: 881.71675ms FAIL (2025-07-26 22:44:20 UTC) orbiter warning: uplink: too many requests Duration: 881.740595ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.926557368s FAIL (2025-07-26 22:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.926580903s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................................X...X...............X..XX..] duration: 16.632322476s FAIL (2025-07-26 22:44:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.632339427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.XXX.....X.........................X..........X.X........X.] duration: 5.792419624s FAIL (2025-07-26 22:44:04 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 5.792439492s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..................................X...X...............X..XX.] duration: 47.531902791s FAIL (2025-07-26 22:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.531926066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX..XXXX..X.X...X..........................................X] duration: 30.236501606s FAIL (2025-07-26 22:43:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ext canceled Jul 26 22:43:12.753291224 (59h7m22.475854264s) [noise=true , address=104.36.82.12:4149 , piece_id=ENMBE46R5IS3RQI47IQHJCRGWPWFFVN6N5K2CP7VVORSQNTAN6NA, seg_pos={0 0}, upload=56735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Jul 26 22:43:12.753338916 (59h7m22.475901946s) [seg_pos={0 0}, upload=56735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:43:12.753420903 (59h7m22.475983943s) [upload=56735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:43:12.753425401 (59h7m22.475988441s) [upload=56735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:43:13.056411010 (59h7m22.77897405s) [upload=56735]: Done uploading segments... Jul 26 22:43:13.056441439 (59h7m22.779004469s) [upload=56735]: Upload finished. err: <nil> Jul 26 22:43:25.427545510 (59h7m35.150108571s) [upload=56740]: Starting upload Jul 26 22:43:25.427645762 (59h7m35.150208812s) [upload=56740]: Done starting upload Jul 26 22:43:25.427765732 (59h7m35.150328812s) [upload=56740]: Uploading segments... Jul 26 22:43:25.427992075 (59h7m35.150555125s) [upload=56740]: Got next segment. Inline: false Jul 26 22:43:25.428037661 (59h7m35.150600721s) [upload=56740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:43:25.428044565 (59h7m35.150607625s) [upload=56740]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:43:25.878521457 (59h7m35.601084497s) [seg_pos={0 0}, upload=56740]: Begin upload segment... Jul 26 22:43:25.878580801 (59h7m35.601143841s) [seg_pos={0 0}, upload=56740]: Done begin upload segment. Jul 26 22:43:25.878601821 (59h7m35.601164861s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878650815 (59h7m35.601213855s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878955819 (59h7m35.601518859s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.878959215 (59h7m35.601522255s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879199194 (59h7m35.601762234s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879202030 (59h7m35.60176507s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879439463 (59h7m35.602002503s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879442830 (59h7m35.60200586s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879668643 (59h7m35.602231683s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879673612 (59h7m35.602236642s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879696055 (59h7m35.602259085s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879897640 (59h7m35.60246068s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.879931285 (59h7m35.602494325s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880122240 (59h7m35.60268527s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880160684 (59h7m35.602723714s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:25.880398408 (59h7m35.602961438s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Uploading piece... Jul 26 22:43:27.554774284 (59h7m37.277337324s) [upload=56740]: Next returned nil segment Jul 26 22:43:27.554794894 (59h7m37.277357924s) [upload=56740]: Waiting for error group managing segments... Jul 26 22:43:27.895438533 (59h7m37.618001573s) [noise=true , address=45.91.116.5:4299 , piece_id=WGH2AM3GOBBZZYPLGWV77RCVWJMZXEL6SLM64W5TTL7GAZONQJGA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.896222523 (59h7m37.618785563s) [noise=true , address=77.90.31.11:4554 , piece_id=CFZCCAVOFU42EK37ILBIVCOVBFPVPJEEDNMWNVN44LWGK3TBGD3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.970692175 (59h7m37.693255215s) [noise=true , address=45.91.116.11:4117 , piece_id=PNAS626CWPDHBDXWRITVFC6MCJPX5CORCBQZWPTMOEXE3QCLHM7A, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.978969372 (59h7m37.701532412s) [noise=true , address=77.90.31.14:4662 , piece_id=ZXD46R2JGXHAVX4HUNW65CXGK44USI5523OQJEND5TUPOIAUT5HQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.987336060 (59h7m37.7098991s) [noise=true , address=45.91.116.14:4191 , piece_id=QCTZICCLXNLDU4MNYXOYZZWJ4Y7M6TRRF7JWO53VTVGOWBFOAYPQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:27.998865999 (59h7m37.72142904s) [noise=true , address=45.91.116.12:4225 , piece_id=ZYL6UCKLXFOO5HTSUV2XT3UO6UQREKLMYUCR5QPX54CS6JDBHK3Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.043962462 (59h7m37.766525502s) [noise=true , address=45.91.116.13:4153 , piece_id=TEEM2NOWFWPDCCFYCDU7IXPLWERZABN6IR2R47UERZEHZGJQXJBQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.054932351 (59h7m37.777495391s) [noise=true , address=45.91.116.7:4009 , piece_id=HKGTR7P7VGUA6BTWGWM2ZI77M344BHW6Z6Y7HB6YNT6NOSTMV63Q, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.076701943 (59h7m37.799264983s) [noise=true , address=104.36.82.3:4767 , piece_id=D3NDRZ4LP2MJQSQDADO4M5WWOOMEWSFRQH75BCSZZYHJ6IXAP4BQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.274988773 (59h7m37.997551813s) [noise=true , address=45.91.116.11:4119 , piece_id=LPNK6UX2U3X2NUW6OXKZYVTEMEX5HKMJ3CWWZ44HUJTVSZRK4XDQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:28.705301275 (59h7m38.427864315s) [noise=true , address=77.90.31.10:4520 , piece_id=PLFGQ7SLBUXDO4P3IWJE7XLAQA2YWCFLUU4MQI7OMS6PCIIYOSQQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:29.535144084 (59h7m39.257707124s) [noise=true , address=77.90.31.18:4807 , piece_id=EOOXTQGEYQSWBF5MN3KVTBMXGCSJ6BXN3YYYZPAYTV37IBBQT7JQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: <nil> Jul 26 22:43:54.781624235 (59h8m4.504187275s) [noise=true , address=77.90.31.12:4592 , piece_id=UXSUM4SQVLNPOWUIEBWCBVT6FEFSRDVL7IYGWBD6G3YUVSKXDSMA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:34962->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781698937 (59h8m4.504261967s) [noise=true , address=104.36.82.8:4969 , piece_id=5A5NMZ5QFPNBVSUYVFSJT52U743QHL7S3STSYMCA4ZBALCEJUVUA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:54606->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781732492 (59h8m4.504295532s) [noise=true , address=104.36.82.8:4971 , piece_id=DBSKY6U6YCAYDEMKNBNQAIBY2KYUQ4XJHNVFVRB67LMY6L7X6EHA, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:51576->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:43:54.781780303 (59h8m4.504343343s) [noise=true , address=104.36.82.14:4223 , piece_id=HT47KG3GKX3KWPEWHUT5TT6FD2NHVPMH4DKMEQSK3A27YJJ3GPSQ, seg_pos={0 0}, upload=56740]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:54.781838375 (59h8m4.504401415s) [seg_pos={0 0}, upload=56740]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:43:55.015418060 (59h8m4.73798116s) [upload=56740]: Done uploading segments... Jul 26 22:43:55.015449080 (59h8m4.73801211s) [upload=56740]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236528155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.X] duration: 1m0.238079386s FAIL (2025-07-26 22:43:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ) [upload=56728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:42:27.471737490 (59h6m37.19430052s) [upload=56728]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:42:27.913703499 (59h6m37.636266539s) [seg_pos={0 0}, upload=56728]: Begin upload segment... Jul 26 22:42:27.913745379 (59h6m37.636308419s) [seg_pos={0 0}, upload=56728]: Done begin upload segment. Jul 26 22:42:27.913766710 (59h6m37.63632975s) [noise=true , address=79.127.201.215:4353 , piece_id=BFGWDQNDDAYLS6QIHFJTREPEJQFQ4NC45IOHFBKLXABF7DKP2CZQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.917828481 (59h6m37.640391521s) [noise=true , address=77.90.31.18:4828 , piece_id=H6IVCTVVU5ICNPNVKAINWXW5KV446TGGHOGJLP5ETZMFOTKC2GVA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.918083569 (59h6m37.640646609s) [noise=true , address=79.127.205.233:4635 , piece_id=WREOG2YJLV77QDR2A3ROPIPE7BWLBJHZEI7EU44N3KS3DTROSSTQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.918286498 (59h6m37.640849528s) [noise=true , address=45.91.116.14:4206 , piece_id=XJVWKZLZN3LT2MQYEXOVT7GLH5VSXKQ3FPP4FQPDNXGPRPIJ5DLA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.918481160 (59h6m37.64104419s) [noise=true , address=45.91.116.13:4175 , piece_id=VQFCTVHE6VXMDFVAKAQIHD6NRXV46BRD4WFCEHBF5VGRX7MIUISQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.918665212 (59h6m37.641228252s) [noise=true , address=45.91.116.9:4267 , piece_id=QXYABSFVEWYR2YATEZE57U6VZRCDGFF6CTCDB2TITJBMLWVBZD7A, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.918930500 (59h6m37.64149353s) [noise=true , address=64.49.13.155:4923 , piece_id=2AEBEVRQ5YTMUHSWMSNLYBE2QBIR75YPK2ACIYWCSNHIZG77NNNA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.919123479 (59h6m37.641686519s) [noise=true , address=156.146.43.225:4448 , piece_id=TZMOQ22KHWD2WW52GDNMILGTLPB7PB7WS7YW6BK3JNTFE4QYALCA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.919305728 (59h6m37.641868768s) [noise=true , address=79.127.220.101:4048 , piece_id=XGREB57WAXFRRWPQF2ZIONTBFBZUIRN465DYZ4A2Z2VFXG5O5PRQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.919539144 (59h6m37.642102184s) [noise=true , address=77.90.31.13:4643 , piece_id=C6GJAGJ3QOKJP63YVQ57CRESOHFMG3SFAQRUMSORS5C4UUZ75KPA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.924696933 (59h6m37.647259973s) [noise=true , address=79.127.219.51:4212 , piece_id=4MGLZ4X36MRKGGZS75AYELADSVXZ2NTOG7CSDJZ6XXF5TXO526MA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.925776218 (59h6m37.648339248s) [noise=true , address=104.36.82.43:4841 , piece_id=2KKECIO4ESYPTINJZKMMACYITOGJAHM3ZINQIQWCEB2HMGGFEODQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.926084998 (59h6m37.648648038s) [noise=true , address=79.127.219.38:4005 , piece_id=MXWDNW3GKHZKXHYEAOIG4RPTUYI5BMVAGQPSCZHAKQEQOJVXNJEA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.926589333 (59h6m37.649152373s) [noise=true , address=77.90.31.19:4847 , piece_id=Z3T75R3VTWMCLWYTM6XBUNIYF2JHY2B5BTU5ITM3ENHFK6KUBKAA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.926896752 (59h6m37.649459792s) [noise=true , address=79.127.205.245:4249 , piece_id=VFEH2EFL6FZJOWH2OBWKO6GGFXSXALPIXGRWK5HRBKKDC7Q37Y7Q, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.927962911 (59h6m37.650525951s) [noise=true , address=64.49.13.138:4482 , piece_id=UJZAR45A7V6YDPXXO6JJPCDSV3YAS3NBYZK4G4MZN2XBUP6KMKJQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.928170028 (59h6m37.650733068s) [noise=true , address=45.91.116.11:4121 , piece_id=4K2KVYY6V6BJBV7ABYVVXHYNP73MM27SGMXUUSQRJWLN3PJ633UA, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:27.928512883 (59h6m37.651075923s) [noise=true , address=79.127.219.48:4090 , piece_id=JFT2VU6GLJS656GLUDNQZTYIJC3HNZR6JWNVLY5JGI7G7UNVO2FQ, seg_pos={0 0}, upload=56728]: Uploading piece... Jul 26 22:42:28.883227028 (59h6m38.605790078s) [noise=true , address=79.127.220.101:4048 , piece_id=XGREB57WAXFRRWPQF2ZIONTBFBZUIRN465DYZ4A2Z2VFXG5O5PRQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:28.947378187 (59h6m38.669941227s) [noise=true , address=64.49.13.155:4923 , piece_id=2AEBEVRQ5YTMUHSWMSNLYBE2QBIR75YPK2ACIYWCSNHIZG77NNNA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z, address:64.49.13.155:4923): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:28.973703335 (59h6m38.696266375s) [noise=true , address=79.127.219.51:4212 , piece_id=4MGLZ4X36MRKGGZS75AYELADSVXZ2NTOG7CSDJZ6XXF5TXO526MA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:28.983350232 (59h6m38.705913272s) [noise=true , address=77.90.31.18:4828 , piece_id=H6IVCTVVU5ICNPNVKAINWXW5KV446TGGHOGJLP5ETZMFOTKC2GVA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:29.038125734 (59h6m38.760688784s) [upload=56728]: Next returned nil segment Jul 26 22:42:29.038143898 (59h6m38.760706938s) [upload=56728]: Waiting for error group managing segments... Jul 26 22:42:29.324459804 (59h6m39.047022844s) [noise=true , address=79.127.205.245:4249 , piece_id=VFEH2EFL6FZJOWH2OBWKO6GGFXSXALPIXGRWK5HRBKKDC7Q37Y7Q, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:29.333683210 (59h6m39.05624625s) [noise=true , address=79.127.205.233:4635 , piece_id=WREOG2YJLV77QDR2A3ROPIPE7BWLBJHZEI7EU44N3KS3DTROSSTQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:29.347613664 (59h6m39.070176704s) [noise=true , address=79.127.219.48:4090 , piece_id=JFT2VU6GLJS656GLUDNQZTYIJC3HNZR6JWNVLY5JGI7G7UNVO2FQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:1osS2pUTTyB9EWHhgp2nKmPbMbezJrQAMKk1JyAEfcg1iBQPAS, address:79.127.219.48:4090): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:29.928914380 (59h6m39.65147742s) [noise=true , address=79.127.219.38:4005 , piece_id=MXWDNW3GKHZKXHYEAOIG4RPTUYI5BMVAGQPSCZHAKQEQOJVXNJEA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:29.967597041 (59h6m39.690160081s) [noise=true , address=45.91.116.11:4121 , piece_id=4K2KVYY6V6BJBV7ABYVVXHYNP73MM27SGMXUUSQRJWLN3PJ633UA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:30.151403045 (59h6m39.873966085s) [noise=true , address=104.36.82.43:4841 , piece_id=2KKECIO4ESYPTINJZKMMACYITOGJAHM3ZINQIQWCEB2HMGGFEODQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:30.298871646 (59h6m40.021434696s) [noise=true , address=64.49.13.138:4482 , piece_id=UJZAR45A7V6YDPXXO6JJPCDSV3YAS3NBYZK4G4MZN2XBUP6KMKJQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:30.979806481 (59h6m40.702369511s) [noise=true , address=45.91.116.9:4267 , piece_id=QXYABSFVEWYR2YATEZE57U6VZRCDGFF6CTCDB2TITJBMLWVBZD7A, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:31.512949519 (59h6m41.235512559s) [noise=true , address=45.91.116.13:4175 , piece_id=VQFCTVHE6VXMDFVAKAQIHD6NRXV46BRD4WFCEHBF5VGRX7MIUISQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:34.541785108 (59h6m44.264348148s) [noise=true , address=156.146.43.225:4448 , piece_id=TZMOQ22KHWD2WW52GDNMILGTLPB7PB7WS7YW6BK3JNTFE4QYALCA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:36.110229536 (59h6m45.832792566s) [noise=true , address=45.91.116.14:4206 , piece_id=XJVWKZLZN3LT2MQYEXOVT7GLH5VSXKQ3FPP4FQPDNXGPRPIJ5DLA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:42:39.401706212 (59h6m49.124269252s) [noise=true , address=77.90.31.19:4847 , piece_id=Z3T75R3VTWMCLWYTM6XBUNIYF2JHY2B5BTU5ITM3ENHFK6KUBKAA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:42:41.783306633 (59h6m51.505869673s) [noise=true , address=79.127.201.215:4353 , piece_id=BFGWDQNDDAYLS6QIHFJTREPEJQFQ4NC45IOHFBKLXABF7DKP2CZQ, seg_pos={0 0}, upload=56728]: Done uploading piece. err: <nil> Jul 26 22:43:26.828635210 (59h7m36.55119825s) [noise=true , address=77.90.31.13:4643 , piece_id=C6GJAGJ3QOKJP63YVQ57CRESOHFMG3SFAQRUMSORS5C4UUZ75KPA, seg_pos={0 0}, upload=56728]: Done uploading piece. err: ecclient: upload failed (node:12qJRodezwvD9yoBUTTiWsqqAbW3YCcZ7DBJEXCqPebkeRkb3QA, address:77.90.31.13:4643): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:43:26.828663784 (59h7m36.551226824s) [seg_pos={0 0}, upload=56728]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:43:27.064504095 (59h7m36.787067135s) [upload=56728]: Done uploading segments... Jul 26 22:43:27.064528242 (59h7m36.787091282s) [upload=56728]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.23810282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.XXX.....X.........................X..........X.X........X] duration: 4.88639764s FAIL (2025-07-26 22:43:22 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 4.886424812s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX..XXXX..X.X...X..........................................] duration: 5.753605652s FAIL (2025-07-26 22:43:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.753656027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...................................X...X...............X..XX] duration: 1m0.001153329s FAIL (2025-07-26 22:43:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001173697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.898080987s FAIL (2025-07-26 22:43:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.898103499s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.XXX.....X.........................X..........X.X........] duration: 5.704577464s FAIL (2025-07-26 22:42:25 UTC) orbiter warning: digest mismatch: after 0b91e0464c41c8a35eaa81ee35c519bb5704a0de679a88cceae86950efb0668e != before d63ea0e90bcf040c30aa7ac6160c8c22d34f24eb6275e1dca16a9061bcb496cd Duration: 5.704620366s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX..XXXX..X.X...X.........................................] duration: 5.846267419s FAIL (2025-07-26 22:42:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.846294791s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.850456274s FAIL (2025-07-26 22:42:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.8504759s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.] duration: 24.252203623s FAIL (2025-07-26 22:42:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 22:40:03.003643879 (59h4m12.726206939s) [upload=56682]: Upload finished. err: <nil> Jul 26 22:40:36.769327702 (59h4m46.491890742s) [upload=56695]: Starting upload Jul 26 22:40:36.769383308 (59h4m46.491946348s) [upload=56695]: Done starting upload Jul 26 22:40:36.769517134 (59h4m46.492080174s) [upload=56695]: Uploading segments... Jul 26 22:40:36.769613889 (59h4m46.492176929s) [upload=56695]: Got next segment. Inline: false Jul 26 22:40:36.769659056 (59h4m46.492222096s) [upload=56695]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:40:36.769664205 (59h4m46.492227245s) [upload=56695]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:40:37.220475988 (59h4m46.943039028s) [seg_pos={0 0}, upload=56695]: Begin upload segment... Jul 26 22:40:37.220519422 (59h4m46.943082462s) [seg_pos={0 0}, upload=56695]: Done begin upload segment. Jul 26 22:40:37.220535112 (59h4m46.943098152s) [noise=true , address=79.127.205.245:4236 , piece_id=EYTKQ26A2MNDGZNETUDY3BIAPALF357IWPMSN3LSPIAJIBUDKSQA, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.220543157 (59h4m46.943106187s) [noise=true , address=77.90.31.19:4849 , piece_id=7KQOBWFJKCNSPVE3M2LQRY5TGDO5OLQK2K4PP5XSJII5XJZZIUGA, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.220868790 (59h4m46.94343183s) [noise=true , address=77.90.31.11:4565 , piece_id=AKN3CLJ53TCPYSCPEZWU462EQBENRWEKIHDN7QQDVU4MTT47FA7A, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.221126022 (59h4m46.943689062s) [noise=true , address=79.127.205.234:4614 , piece_id=ZHJBSS3AYKGSXEE5WNNQT4W3R3LOFD5JBKYIMJ2UTQOQ3E3XMGYQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.221374638 (59h4m46.943937678s) [noise=true , address=79.127.219.51:4210 , piece_id=2ECX6WHOF76OEHRMPULRHHTS2GUR5ZYK3IUZBRBW7COXJKAPXTFQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.221466353 (59h4m46.944029383s) [noise=true , address=79.127.219.38:4001 , piece_id=3WPGP2FP4YUQZ7NQ7VSJZIOMLKX6THE7SKP7IELMBN4ZS4UH3NPQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.221736760 (59h4m46.944299791s) [noise=true , address=77.90.31.10:4525 , piece_id=DTMU6MGMHIHAV6TJSK7BLEGXFNKENWCM3U5ZDEGDCNYVQBZCCQ4Q, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222010675 (59h4m46.944573715s) [noise=true , address=45.91.116.11:4121 , piece_id=GAXOSWZEDSG7LWCD7LXRBEV56H6TLLKOLJW7O5UDPMAEDI6KWJQA, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222048237 (59h4m46.944611267s) [noise=true , address=77.90.31.18:4824 , piece_id=2LHGAMKKFWP5HDXVY6KU5UHGRZ6TUGAUNWJJ4GYAZTVWJZNZJ2QA, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222216178 (59h4m46.944779218s) [noise=true , address=79.127.219.47:4048 , piece_id=Q7RQRQACY5QKLUCCKXIEK5GISYLBWXB7GZGBHVOYN3G3SECRJZUQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222261374 (59h4m46.944824404s) [noise=true , address=77.90.31.14:4684 , piece_id=HJYKLPL3FYFJ7IUPOJOJ22JRKGJBQCMMA7B4F2U3I4DJM73FH63Q, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222454433 (59h4m46.945017474s) [noise=true , address=79.127.205.235:4690 , piece_id=K32AX463EOFBKTX224YIWXN7ZV2V7HALBWI6GHFWXKSLISCJLP7A, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222643455 (59h4m46.945206495s) [noise=true , address=156.146.43.225:4449 , piece_id=DAGUQ7NPE3UVW7OBXCEM525HEDFAIXEQZR6FR7SC5ZSBNB43FFBQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222831766 (59h4m46.945394806s) [noise=true , address=77.90.31.12:4611 , piece_id=3NY73GDUZWBVQLKICNPZUA57HCNGO7ZVK34C27W75YCBR3QAAOVQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.222908763 (59h4m46.945471793s) [noise=true , address=193.149.164.143:4702 , piece_id=PBVLO7H2MN3OKNKHNHXPR4TIWC7ENUZZJEITP4LCLRB5FBCRC54A, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.223142770 (59h4m46.945705811s) [noise=true , address=45.91.116.14:4222 , piece_id=WKQPWCLLHHEKQGW4AKBQGKTALFMH24O3K72UPKU2FP24CHMEEP7Q, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.223331111 (59h4m46.945894151s) [noise=true , address=79.127.201.215:4353 , piece_id=LJVXPGL7UIDCCHLIYH3JNKOJL336IAOCLNJ3IYZQQRGGRH2CKNAQ, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:37.223384834 (59h4m46.945947874s) [noise=true , address=79.127.219.48:4090 , piece_id=ME5PWG46MPYH44KPYPSKWM6BTGMV6OSQCZPPUDTNEBTVNN7MNXVA, seg_pos={0 0}, upload=56695]: Uploading piece... Jul 26 22:40:38.267093881 (59h4m47.989656931s) [noise=true , address=79.127.219.51:4210 , piece_id=2ECX6WHOF76OEHRMPULRHHTS2GUR5ZYK3IUZBRBW7COXJKAPXTFQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:38.290663667 (59h4m48.013226717s) [noise=true , address=77.90.31.19:4849 , piece_id=7KQOBWFJKCNSPVE3M2LQRY5TGDO5OLQK2K4PP5XSJII5XJZZIUGA, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V, address:77.90.31.19:4849): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:38.324426795 (59h4m48.046989835s) [upload=56695]: Next returned nil segment Jul 26 22:40:38.324444850 (59h4m48.04700789s) [upload=56695]: Waiting for error group managing segments... Jul 26 22:40:38.645578401 (59h4m48.368141442s) [noise=true , address=79.127.205.235:4690 , piece_id=K32AX463EOFBKTX224YIWXN7ZV2V7HALBWI6GHFWXKSLISCJLP7A, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:38.660037525 (59h4m48.382600565s) [noise=true , address=79.127.205.245:4236 , piece_id=EYTKQ26A2MNDGZNETUDY3BIAPALF357IWPMSN3LSPIAJIBUDKSQA, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:38.918540457 (59h4m48.641103497s) [noise=true , address=79.127.205.234:4614 , piece_id=ZHJBSS3AYKGSXEE5WNNQT4W3R3LOFD5JBKYIMJ2UTQOQ3E3XMGYQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:39.136773930 (59h4m48.85933698s) [noise=true , address=77.90.31.11:4565 , piece_id=AKN3CLJ53TCPYSCPEZWU462EQBENRWEKIHDN7QQDVU4MTT47FA7A, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:39.152528884 (59h4m48.875091924s) [noise=true , address=77.90.31.10:4525 , piece_id=DTMU6MGMHIHAV6TJSK7BLEGXFNKENWCM3U5ZDEGDCNYVQBZCCQ4Q, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:39.194478465 (59h4m48.917041505s) [noise=true , address=77.90.31.12:4611 , piece_id=3NY73GDUZWBVQLKICNPZUA57HCNGO7ZVK34C27W75YCBR3QAAOVQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:39.209572645 (59h4m48.932135685s) [noise=true , address=79.127.219.38:4001 , piece_id=3WPGP2FP4YUQZ7NQ7VSJZIOMLKX6THE7SKP7IELMBN4ZS4UH3NPQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:39.227835487 (59h4m48.950398527s) [noise=true , address=77.90.31.18:4824 , piece_id=2LHGAMKKFWP5HDXVY6KU5UHGRZ6TUGAUNWJJ4GYAZTVWJZNZJ2QA, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:40.209946910 (59h4m49.93250995s) [noise=true , address=79.127.219.48:4090 , piece_id=ME5PWG46MPYH44KPYPSKWM6BTGMV6OSQCZPPUDTNEBTVNN7MNXVA, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:41.070009408 (59h4m50.792572448s) [noise=true , address=79.127.219.47:4048 , piece_id=Q7RQRQACY5QKLUCCKXIEK5GISYLBWXB7GZGBHVOYN3G3SECRJZUQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:41.216409073 (59h4m50.938972113s) [noise=true , address=193.149.164.143:4702 , piece_id=PBVLO7H2MN3OKNKHNHXPR4TIWC7ENUZZJEITP4LCLRB5FBCRC54A, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Jul 26 22:40:41.937824730 (59h4m51.66038777s) [noise=true , address=156.146.43.225:4449 , piece_id=DAGUQ7NPE3UVW7OBXCEM525HEDFAIXEQZR6FR7SC5ZSBNB43FFBQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:40:43.093676389 (59h4m52.816239429s) [noise=true , address=45.91.116.11:4121 , piece_id=GAXOSWZEDSG7LWCD7LXRBEV56H6TLLKOLJW7O5UDPMAEDI6KWJQA, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:10.882747743 (59h5m20.605310783s) [noise=true , address=79.127.201.215:4353 , piece_id=LJVXPGL7UIDCCHLIYH3JNKOJL336IAOCLNJ3IYZQQRGGRH2CKNAQ, seg_pos={0 0}, upload=56695]: Done uploading piece. err: <nil> Jul 26 22:41:13.919765236 (59h5m23.642328266s) [noise=true , address=77.90.31.14:4684 , piece_id=HJYKLPL3FYFJ7IUPOJOJ22JRKGJBQCMMA7B4F2U3I4DJM73FH63Q, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m, address:77.90.31.14:4684): protocol: storage node overloaded, request limit: 2000 Jul 26 22:41:36.123313348 (59h5m45.845876378s) [noise=true , address=45.91.116.14:4222 , piece_id=WKQPWCLLHHEKQGW4AKBQGKTALFMH24O3K72UPKU2FP24CHMEEP7Q, seg_pos={0 0}, upload=56695]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:41:36.123338496 (59h5m45.845901536s) [seg_pos={0 0}, upload=56695]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 22:41:36.355647551 (59h5m46.078210591s) [upload=56695]: Done uploading segments... Jul 26 22:41:36.355680884 (59h5m46.078243965s) [upload=56695]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.25222883s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)