Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.849941744s FAIL (2025-07-27 00:18:11 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.849970518s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X...] duration: 6.538882597s FAIL (2025-07-27 00:18:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.538908527s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....X] duration: 30.235111243s FAIL (2025-07-27 00:18:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 038 (60h40m54.227618078s) [upload=58197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:16:44.795161809 (60h40m54.517724839s) [upload=58197]: Done uploading segments... Jul 27 00:16:44.795179844 (60h40m54.517742884s) [upload=58197]: Upload finished. err: <nil> Jul 27 00:17:33.962231008 (60h41m43.684794048s) [upload=58209]: Starting upload Jul 27 00:17:33.962310229 (60h41m43.684873269s) [upload=58209]: Done starting upload Jul 27 00:17:33.962390943 (60h41m43.684953983s) [upload=58209]: Uploading segments... Jul 27 00:17:33.962576528 (60h41m43.685139558s) [upload=58209]: Got next segment. Inline: false Jul 27 00:17:33.962616475 (60h41m43.685179515s) [upload=58209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:17:33.962621615 (60h41m43.685184655s) [upload=58209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:17:34.194429976 (60h41m43.916993016s) [seg_pos={0 0}, upload=58209]: Begin upload segment... Jul 27 00:17:34.194474481 (60h41m43.917037521s) [seg_pos={0 0}, upload=58209]: Done begin upload segment. Jul 27 00:17:34.194500090 (60h41m43.91706314s) [noise=true , address=77.90.31.10:4520 , piece_id=26JWCEJCLZYB73XLT2XIO5ONDV6MGCKAUD55ENT7D5YJQ4GCQUEA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194504578 (60h41m43.917067619s) [noise=true , address=104.36.82.2:4731 , piece_id=DT7ALGELQZCFIP6Z7U7VOE5G5623ULD6PSVI7LFEQEWU2T27V5EQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194886700 (60h41m43.91744974s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZFP2I5NAHKUVWZW7EYJOOYF32IO3T2R5CE2ZYNEYUEIXRCOOA4A, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.194934141 (60h41m43.917497181s) [noise=true , address=77.90.31.17:4771 , piece_id=QTMLXEIMKZ74EYLQVQIV4WNTQHO2RNSYZLJ5QUUZWD446ROSQ5WA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195273580 (60h41m43.91783662s) [noise=true , address=45.91.116.14:4189 , piece_id=GQKKQL2B52K2CNLYOLAXUDWZO73VFEGONGAEOEDCQMTT3BPIWZ3Q, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195591829 (60h41m43.918154879s) [noise=true , address=77.90.31.16:4734 , piece_id=V24CW3MXIP73N7IWYESHDAV4QXRZKYSZ2U4D3FQELK4O4XEBD5HA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195625543 (60h41m43.918188583s) [noise=true , address=77.90.31.13:4626 , piece_id=MIYCINRM67BQB3KRIAAF2WUN4B24BZKWAW4W2G6YPTRVQKYE4O7Q, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.195886343 (60h41m43.918449623s) [noise=true , address=77.90.31.10:4517 , piece_id=LTIBGQWRDGTW2LPMFZNDTBP236TLVTOZGA4TMCJQL6LLJSRAXYTQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.196281128 (60h41m43.918844168s) [noise=true , address=77.90.31.14:4663 , piece_id=JGRSTHDJFOA3TFWRLA62DCX4JJJTJASM6NFN663UK2CA22Z7YGSA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197037776 (60h41m43.919600816s) [noise=true , address=45.91.116.5:4299 , piece_id=BQM6HZK2CNCV23SEXCBDJGYCICKXOAIXAVDBSLGYI56ZE7KKCDIA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197257166 (60h41m43.919820206s) [noise=true , address=45.91.116.11:4118 , piece_id=H2UNNR2QQ4ZFRLCC3D4Z4WSYC4IYHBBVQYG6FIMSIQV4XY2QL4TQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197455756 (60h41m43.920018796s) [noise=true , address=45.91.116.11:4119 , piece_id=VSLZZ77LFNQQKJJWFIHAGMTAJUNITHT3RK3DE7P6SJXQLDMSS4FQ, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197653474 (60h41m43.920216514s) [noise=true , address=45.91.116.10:4081 , piece_id=37OMV255QGHBEO7FEHMRL44LQYY4PKNNQVYIRVWWQJPN52P5CYIA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.197888202 (60h41m43.920451242s) [noise=true , address=45.91.116.12:4227 , piece_id=WO6HD4FZWCTXBMM2LQC7GVNTZVNAPD4RJ64L5WIAFWNBIZU245MA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.198098745 (60h41m43.920661786s) [noise=true , address=104.36.82.8:4970 , piece_id=DBNR2K5WQVQIIT5XDMSXFDU5YBH3PUVWZKLA5M3I2HEFLACDM65A, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:34.198294289 (60h41m43.92085733s) [noise=true , address=104.36.82.8:4971 , piece_id=MZ3CIGVDC7LVVZNTNROF6OEQ5K7NTCCR5GT7NPLUOIYMPNZFZ5UA, seg_pos={0 0}, upload=58209]: Uploading piece... Jul 27 00:17:35.949021956 (60h41m45.671584996s) [upload=58209]: Next returned nil segment Jul 27 00:17:35.949041252 (60h41m45.671604292s) [upload=58209]: Waiting for error group managing segments... Jul 27 00:17:36.226962519 (60h41m45.949525559s) [noise=true , address=77.90.31.16:4734 , piece_id=V24CW3MXIP73N7IWYESHDAV4QXRZKYSZ2U4D3FQELK4O4XEBD5HA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.296215582 (60h41m46.018778622s) [noise=true , address=77.90.31.14:4663 , piece_id=JGRSTHDJFOA3TFWRLA62DCX4JJJTJASM6NFN663UK2CA22Z7YGSA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.305929036 (60h41m46.028492076s) [noise=true , address=45.91.116.11:4118 , piece_id=H2UNNR2QQ4ZFRLCC3D4Z4WSYC4IYHBBVQYG6FIMSIQV4XY2QL4TQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.311637579 (60h41m46.034200619s) [noise=true , address=45.91.116.10:4081 , piece_id=37OMV255QGHBEO7FEHMRL44LQYY4PKNNQVYIRVWWQJPN52P5CYIA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.361868666 (60h41m46.084431706s) [noise=true , address=45.91.116.5:4299 , piece_id=BQM6HZK2CNCV23SEXCBDJGYCICKXOAIXAVDBSLGYI56ZE7KKCDIA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.366642170 (60h41m46.0892052s) [noise=true , address=45.91.116.11:4119 , piece_id=VSLZZ77LFNQQKJJWFIHAGMTAJUNITHT3RK3DE7P6SJXQLDMSS4FQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.428801869 (60h41m46.151364909s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZFP2I5NAHKUVWZW7EYJOOYF32IO3T2R5CE2ZYNEYUEIXRCOOA4A, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.429246590 (60h41m46.15180963s) [noise=true , address=45.91.116.14:4189 , piece_id=GQKKQL2B52K2CNLYOLAXUDWZO73VFEGONGAEOEDCQMTT3BPIWZ3Q, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:36.462050493 (60h41m46.184613533s) [noise=true , address=45.91.116.12:4227 , piece_id=WO6HD4FZWCTXBMM2LQC7GVNTZVNAPD4RJ64L5WIAFWNBIZU245MA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:17:37.736988755 (60h41m47.459551795s) [noise=true , address=77.90.31.13:4626 , piece_id=MIYCINRM67BQB3KRIAAF2WUN4B24BZKWAW4W2G6YPTRVQKYE4O7Q, seg_pos={0 0}, upload=58209]: Done uploading piece. err: <nil> Jul 27 00:18:03.315118280 (60h42m13.03768132s) [noise=true , address=104.36.82.8:4970 , piece_id=DBNR2K5WQVQIIT5XDMSXFDU5YBH3PUVWZKLA5M3I2HEFLACDM65A, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:44300->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315160911 (60h42m13.037723951s) [noise=true , address=104.36.82.8:4971 , piece_id=MZ3CIGVDC7LVVZNTNROF6OEQ5K7NTCCR5GT7NPLUOIYMPNZFZ5UA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:57224->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315259931 (60h42m13.037822971s) [noise=true , address=77.90.31.17:4771 , piece_id=QTMLXEIMKZ74EYLQVQIV4WNTQHO2RNSYZLJ5QUUZWD446ROSQ5WA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:55870->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315411350 (60h42m13.03797439s) [noise=true , address=77.90.31.10:4520 , piece_id=26JWCEJCLZYB73XLT2XIO5ONDV6MGCKAUD55ENT7D5YJQ4GCQUEA, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:53376->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315444585 (60h42m13.038007625s) [noise=true , address=104.36.82.2:4731 , piece_id=DT7ALGELQZCFIP6Z7U7VOE5G5623ULD6PSVI7LFEQEWU2T27V5EQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:36034->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:18:03.315573330 (60h42m13.038136371s) [noise=true , address=77.90.31.10:4517 , piece_id=LTIBGQWRDGTW2LPMFZNDTBP236TLVTOZGA4TMCJQL6LLJSRAXYTQ, seg_pos={0 0}, upload=58209]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:38714->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:18:03.315606073 (60h42m13.038169103s) [seg_pos={0 0}, upload=58209]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:18:03.548994879 (60h42m13.271557929s) [upload=58209]: Done uploading segments... Jul 27 00:18:03.549025557 (60h42m13.271588627s) [upload=58209]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.235134668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.X] duration: 30.260495401s FAIL (2025-07-27 00:17:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.260508746s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.851260248s FAIL (2025-07-27 00:17:41 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.851295877s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239180401s FAIL (2025-07-27 00:17:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ABL2LSY3XMCZNUXCSMFO7TZZLQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.326910813 (60h40m43.049473854s) [noise=true , address=104.36.82.52:4098 , piece_id=3V5VPN5QFWTKSMYEBONIQVBFXDPONMIPV7AW5ADMZ5TFSDHYCZFA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327042626 (60h40m43.049605666s) [noise=true , address=77.90.31.15:4731 , piece_id=3BXDCVJYGS6AFH2PRXS6QXN72PX2OJHUPNCVA33MK5SOEMZZPIFA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327308224 (60h40m43.049871264s) [noise=true , address=104.36.82.47:4546 , piece_id=ZMYG6R6HYPQNWFQXN57WU6RK3IJRUPG5VNU7XD4JPVFUJTOYU4UQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327362669 (60h40m43.049925699s) [noise=true , address=79.127.205.246:4275 , piece_id=X5EJR5TKWOTWRMDTJDRXB5B6HGVAPNF3E43OIPEQ2NQBZTTCMEHA, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327620542 (60h40m43.050183582s) [noise=true , address=104.36.82.53:4141 , piece_id=TKMODPYCU2JN5CWQMLMJVTHTFKIZGRODE3BYDDTINQPTCFV2HCDQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327637194 (60h40m43.050200224s) [noise=true , address=79.127.205.234:4620 , piece_id=RQKRPA4AG7IQ5FHBHOD73VFBVRCPT5ERGVTJMTC6LAG65Z46HUIQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327868075 (60h40m43.050431115s) [noise=true , address=79.127.220.101:4048 , piece_id=YDZEOSWHU5VKEQUX4KWVEBOBWQDPQD6EHLPOSPP5FPBM5E6TE34Q, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.327887683 (60h40m43.050450723s) [noise=true , address=77.90.31.21:4908 , piece_id=7POCZM436STIDLGT3CUAUTUQMMREXJWRHGQH6OZQBLTZASLRUUXQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328181115 (60h40m43.050744155s) [noise=true , address=104.36.82.12:4181 , piece_id=FIKOB2EPEKOOMTDFPSYQWW5KYAXVJO7COD6BINI2EZCKI27QFJQQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328215540 (60h40m43.05077857s) [noise=true , address=104.36.82.21:4507 , piece_id=QEMOQHL74KEZ2VCANXRT2LXYZVIZR2EKY2VEJJQLKJHRUQ3C653A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328433457 (60h40m43.050996497s) [noise=true , address=79.127.205.245:4241 , piece_id=74QK5TUK2GVVFFVTEXLP5DMF7ZF554VWE3PJIKGZHTKHEJZFZFYQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328632177 (60h40m43.051195217s) [noise=true , address=104.36.82.55:4199 , piece_id=QZ4MNNJWW3SUSSLDQNQEHQ6CH2XN5GOPQWGLQ3CDVJ3KBECUBGQQ, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.328686642 (60h40m43.051249672s) [noise=true , address=104.36.82.49:4990 , piece_id=TVMAWIQA3XXIPT26SRWADBNBFCEH457ORJVLAAA34XVA36OXI27A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.329217747 (60h40m43.051780828s) [noise=true , address=77.90.31.10:4545 , piece_id=KAIY7BGG2KLVBUEZ2DVBBVJSVHFIOJZDOGAR7RBE52QNAOZD5C2A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:33.329740387 (60h40m43.052303447s) [noise=true , address=79.127.201.214:4321 , piece_id=LCS4JM76YANYBADJYD4SHZCTGGA3Z5YHTGRW5JYM5EAAALK2NZ6A, seg_pos={0 0}, upload=58191]: Uploading piece... Jul 27 00:16:34.428894880 (60h40m44.15145792s) [upload=58191]: Next returned nil segment Jul 27 00:16:34.428915189 (60h40m44.151478229s) [upload=58191]: Waiting for error group managing segments... Jul 27 00:16:34.733722888 (60h40m44.456285928s) [noise=true , address=79.127.205.245:4241 , piece_id=74QK5TUK2GVVFFVTEXLP5DMF7ZF554VWE3PJIKGZHTKHEJZFZFYQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.734393011 (60h40m44.456956051s) [noise=true , address=79.127.205.234:4620 , piece_id=RQKRPA4AG7IQ5FHBHOD73VFBVRCPT5ERGVTJMTC6LAG65Z46HUIQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.737835798 (60h40m44.460398838s) [noise=true , address=79.127.205.233:4653 , piece_id=7E3XX3YHGIBK4Y5A37ELQREDTTK6ZAP34C45RE3B5BEKBGZEACOQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:34.790383105 (60h40m44.512946145s) [noise=true , address=79.127.205.246:4275 , piece_id=X5EJR5TKWOTWRMDTJDRXB5B6HGVAPNF3E43OIPEQ2NQBZTTCMEHA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:37.963538347 (60h40m47.686101387s) [noise=true , address=77.90.31.18:4832 , piece_id=MG4YIJP563M47RJLS32ESPSFKAABL2LSY3XMCZNUXCSMFO7TZZLQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:38.305732748 (60h40m48.028295788s) [noise=true , address=79.127.201.214:4321 , piece_id=LCS4JM76YANYBADJYD4SHZCTGGA3Z5YHTGRW5JYM5EAAALK2NZ6A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:38.726111399 (60h40m48.448674429s) [noise=true , address=79.127.220.101:4048 , piece_id=YDZEOSWHU5VKEQUX4KWVEBOBWQDPQD6EHLPOSPP5FPBM5E6TE34Q, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:16:50.140254691 (60h40m59.862817731s) [noise=true , address=77.90.31.15:4731 , piece_id=3BXDCVJYGS6AFH2PRXS6QXN72PX2OJHUPNCVA33MK5SOEMZZPIFA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: <nil> Jul 27 00:17:32.242806642 (60h41m41.965369683s) [noise=true , address=77.90.31.10:4545 , piece_id=KAIY7BGG2KLVBUEZ2DVBBVJSVHFIOJZDOGAR7RBE52QNAOZD5C2A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1ZFFijCUFET1ShEJuav6uAzFhfraCEcr6seuySYN9ipn4wsZGX, address:77.90.31.10:4545): protocol: write tcp 188.166.225.170:56242->77.90.31.10:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242864313 (60h41m41.965427353s) [noise=true , address=104.36.82.47:4546 , piece_id=ZMYG6R6HYPQNWFQXN57WU6RK3IJRUPG5VNU7XD4JPVFUJTOYU4UQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242896165 (60h41m41.965459205s) [noise=true , address=104.36.82.21:4507 , piece_id=QEMOQHL74KEZ2VCANXRT2LXYZVIZR2EKY2VEJJQLKJHRUQ3C653A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242937413 (60h41m41.965500453s) [noise=true , address=104.36.82.55:4199 , piece_id=QZ4MNNJWW3SUSSLDQNQEHQ6CH2XN5GOPQWGLQ3CDVJ3KBECUBGQQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.242971909 (60h41m41.965534949s) [noise=true , address=77.90.31.21:4908 , piece_id=7POCZM436STIDLGT3CUAUTUQMMREXJWRHGQH6OZQBLTZASLRUUXQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: read tcp 188.166.225.170:34178->77.90.31.21:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243018638 (60h41m41.965581668s) [noise=true , address=104.36.82.12:4181 , piece_id=FIKOB2EPEKOOMTDFPSYQWW5KYAXVJO7COD6BINI2EZCKI27QFJQQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1XH92Sdi4ZiRg1iic5A9Mp8RyKZsk9WfVQHLoQ8t9Vvk2Lyxwh, address:104.36.82.12:4181): protocol: write tcp 188.166.225.170:52330->104.36.82.12:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243051611 (60h41m41.965614651s) [noise=true , address=104.36.82.52:4098 , piece_id=3V5VPN5QFWTKSMYEBONIQVBFXDPONMIPV7AW5ADMZ5TFSDHYCZFA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1SoFNdwfh4p5tXUkpfjQaev6tdA2ZdhsXUFvZNBsy5QrJ9zdqb, address:104.36.82.52:4098): protocol: write tcp 188.166.225.170:41640->104.36.82.52:4098: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243095696 (60h41m41.965658736s) [noise=true , address=104.36.82.46:4509 , piece_id=TW4DDBQPHTSLLCULGYAMROCPWSKWNG5D57MA4L62W2XQ5GTQNBNA, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243113570 (60h41m41.9656766s) [noise=true , address=104.36.82.53:4141 , piece_id=TKMODPYCU2JN5CWQMLMJVTHTFKIZGRODE3BYDDTINQPTCFV2HCDQ, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:17:32.243422882 (60h41m41.965985932s) [noise=true , address=104.36.82.49:4990 , piece_id=TVMAWIQA3XXIPT26SRWADBNBFCEH457ORJVLAAA34XVA36OXI27A, seg_pos={0 0}, upload=58191]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:104.36.82.49:4990): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:17:32.243446277 (60h41m41.966009317s) [seg_pos={0 0}, upload=58191]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:17:32.480047178 (60h41m42.202610218s) [upload=58191]: Done uploading segments... Jul 27 00:17:32.480072977 (60h41m42.202636017s) [upload=58191]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.239192936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..XXX..X.....X......X...XX..XX.X.XX...X...............XX....] duration: 40.335758121s FAIL (2025-07-27 00:17:17 UTC) orbiter warning: context deadline exceeded Duration: 40.335797316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X..] duration: 6.178601714s FAIL (2025-07-27 00:16:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178623836s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.883589984s FAIL (2025-07-27 00:16:53 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.883637044s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX....] duration: 6.098821439s FAIL (2025-07-27 00:16:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.098870504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX.] duration: 5.747576646s FAIL (2025-07-27 00:16:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.747602145s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233126355s FAIL (2025-07-27 00:16:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 219.47:4048 , piece_id=4TA2BB4WQ5BMKHMNOHM2BNTHSBGKI7LLFPYYIFQ2JXWLXAUSDG6Q, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.873863382 (60h39m42.596426422s) [noise=true , address=79.127.220.101:4041 , piece_id=QRKKZEGZOPIP5TM5BKGTJCYVLCQXMPM33QFSSE3FQROV6N25VF2A, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.874265492 (60h39m42.596828532s) [noise=true , address=79.127.205.233:4652 , piece_id=722SYZCW35YCF34WPZ5J2KV2BIMK3INZVYIIU52RU25LONSSODIQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.874333141 (60h39m42.596896182s) [noise=true , address=104.36.82.55:4201 , piece_id=S6JUAWK2XFUZ3HACMXCCOA2MHG2Q6FWVRAN3QY24JS2KDNJISDWA, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.874497385 (60h39m42.597060425s) [noise=true , address=79.127.205.246:4288 , piece_id=XNWRCKAEUKXBFP6GQZRPBKA7ULMQ5J54JSORW4FRAAWWT2NVYF3Q, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.874642965 (60h39m42.597206005s) [noise=true , address=77.90.31.12:4597 , piece_id=YXV6CTCGTFJBCF2BADYXNGO6IOPL24WJBIY2SBE7CU3Q2YOKJAJA, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.874755129 (60h39m42.597318169s) [noise=true , address=104.36.82.52:4101 , piece_id=O25AMZC4HG6U53N6AJPKBL466K4QUVA7W6L4VBEW6H4VTO32IU5Q, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.875049202 (60h39m42.597612232s) [noise=true , address=79.127.201.214:4321 , piece_id=ZBI6V2MERWY7RQT233NA2IDVP4EZ6ZP7WPOXTX5DKFVDBNOEBS7A, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.875522998 (60h39m42.598086038s) [noise=true , address=79.127.205.245:4250 , piece_id=OBQFCKVUDOE2QFQQ2YHXSPWB7EZZEDFBVTMEPGMSYTVQF6WWRJ2A, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.875900420 (60h39m42.59846346s) [noise=true , address=79.127.205.234:4619 , piece_id=4Z6GMHOMA5R4UWXSZXEDNCLJCKD34MCD77J77UAUP5LFQ32PPTNQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.878053851 (60h39m42.600616891s) [noise=true , address=64.49.13.138:4483 , piece_id=34WQMGERYQPURLYHVFYIJ7YWZZWOTWVWHHOCKBMAT65UXOCAJRGA, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.879086706 (60h39m42.601649736s) [noise=true , address=104.36.82.44:4899 , piece_id=2YJUBSKVETPDBS2XTHZZMWFSD6YIAJAOUWOZV4HU2ISELCINESZQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.879506771 (60h39m42.602069811s) [noise=true , address=104.36.82.47:4526 , piece_id=DB6PLAJPYCXQ3WRWRBA3FJKXY6IPCBYUPJUJOW6R53WTUTUCYEIQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.879763121 (60h39m42.602326161s) [noise=true , address=104.36.82.46:4490 , piece_id=EXSRAMENY67LORVDNIKXY7OEFA5SBCJKS4PGDMZZKFJS7EOETFMA, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.880211058 (60h39m42.602774098s) [noise=true , address=104.36.82.54:4183 , piece_id=DSL7CU7TCM5NYRIGREEB7NT23TJTIUYOVVV2ZCNEVL3CVMI5MLKA, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.880365974 (60h39m42.602929014s) [noise=true , address=79.127.219.38:4005 , piece_id=NKB2DBUAFRFC3RXP5F2E5XTJHA6MMDU3YY5ZD6BS63AEZUIJFZFQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.880682209 (60h39m42.603245249s) [noise=true , address=79.127.205.235:4686 , piece_id=SRY6SOY6UPFWBJRJ6C3UJDGPG5YOIQ57MFXU3THMQ6CSEJ4YK3GQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:32.880899966 (60h39m42.603462996s) [noise=true , address=79.127.201.215:4334 , piece_id=FRA5FHJ2X3CVMN76BRITA5W4V7MFL2Z3E4NIXKNTUK6F7MHMQ4WQ, seg_pos={0 0}, upload=58175]: Uploading piece... Jul 27 00:15:33.892993735 (60h39m43.615556775s) [noise=true , address=79.127.201.214:4321 , piece_id=ZBI6V2MERWY7RQT233NA2IDVP4EZ6ZP7WPOXTX5DKFVDBNOEBS7A, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 27 00:15:33.963418378 (60h39m43.685981428s) [upload=58175]: Next returned nil segment Jul 27 00:15:33.963433017 (60h39m43.685996067s) [upload=58175]: Waiting for error group managing segments... Jul 27 00:15:34.080272315 (60h39m43.802835355s) [noise=true , address=79.127.219.47:4048 , piece_id=4TA2BB4WQ5BMKHMNOHM2BNTHSBGKI7LLFPYYIFQ2JXWLXAUSDG6Q, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:15:34.250409409 (60h39m43.972972449s) [noise=true , address=79.127.205.245:4250 , piece_id=OBQFCKVUDOE2QFQQ2YHXSPWB7EZZEDFBVTMEPGMSYTVQF6WWRJ2A, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.288083771 (60h39m44.010646811s) [noise=true , address=79.127.205.246:4288 , piece_id=XNWRCKAEUKXBFP6GQZRPBKA7ULMQ5J54JSORW4FRAAWWT2NVYF3Q, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.294874053 (60h39m44.017437093s) [noise=true , address=79.127.205.234:4619 , piece_id=4Z6GMHOMA5R4UWXSZXEDNCLJCKD34MCD77J77UAUP5LFQ32PPTNQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.326999378 (60h39m44.049562408s) [noise=true , address=79.127.205.235:4686 , piece_id=SRY6SOY6UPFWBJRJ6C3UJDGPG5YOIQ57MFXU3THMQ6CSEJ4YK3GQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.344853148 (60h39m44.067416188s) [noise=true , address=79.127.205.233:4652 , piece_id=722SYZCW35YCF34WPZ5J2KV2BIMK3INZVYIIU52RU25LONSSODIQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.856527966 (60h39m44.579091006s) [noise=true , address=79.127.219.38:4005 , piece_id=NKB2DBUAFRFC3RXP5F2E5XTJHA6MMDU3YY5ZD6BS63AEZUIJFZFQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:34.882838676 (60h39m44.605401716s) [noise=true , address=79.127.201.215:4334 , piece_id=FRA5FHJ2X3CVMN76BRITA5W4V7MFL2Z3E4NIXKNTUK6F7MHMQ4WQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:38.048542990 (60h39m47.77110602s) [noise=true , address=64.49.13.138:4483 , piece_id=34WQMGERYQPURLYHVFYIJ7YWZZWOTWVWHHOCKBMAT65UXOCAJRGA, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:15:38.088068173 (60h39m47.810631213s) [noise=true , address=79.127.220.101:4041 , piece_id=QRKKZEGZOPIP5TM5BKGTJCYVLCQXMPM33QFSSE3FQROV6N25VF2A, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:16:24.371698602 (60h40m34.094261632s) [noise=true , address=77.90.31.12:4597 , piece_id=YXV6CTCGTFJBCF2BADYXNGO6IOPL24WJBIY2SBE7CU3Q2YOKJAJA, seg_pos={0 0}, upload=58175]: Done uploading piece. err: <nil> Jul 27 00:16:31.786540372 (60h40m41.509103402s) [noise=true , address=104.36.82.52:4101 , piece_id=O25AMZC4HG6U53N6AJPKBL466K4QUVA7W6L4VBEW6H4VTO32IU5Q, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786626707 (60h40m41.509189747s) [noise=true , address=104.36.82.44:4899 , piece_id=2YJUBSKVETPDBS2XTHZZMWFSD6YIAJAOUWOZV4HU2ISELCINESZQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:104.36.82.44:4899): protocol: write tcp 188.166.225.170:44802->104.36.82.44:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786661935 (60h40m41.509224975s) [noise=true , address=104.36.82.47:4526 , piece_id=DB6PLAJPYCXQ3WRWRBA3FJKXY6IPCBYUPJUJOW6R53WTUTUCYEIQ, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:57562->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786695539 (60h40m41.509258579s) [noise=true , address=104.36.82.46:4490 , piece_id=EXSRAMENY67LORVDNIKXY7OEFA5SBCJKS4PGDMZZKFJS7EOETFMA, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: write tcp 188.166.225.170:47560->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786706620 (60h40m41.50926966s) [noise=true , address=104.36.82.54:4183 , piece_id=DSL7CU7TCM5NYRIGREEB7NT23TJTIUYOVVV2ZCNEVL3CVMI5MLKA, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:50230->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786738741 (60h40m41.509301781s) [noise=true , address=104.36.82.55:4201 , piece_id=S6JUAWK2XFUZ3HACMXCCOA2MHG2Q6FWVRAN3QY24JS2KDNJISDWA, seg_pos={0 0}, upload=58175]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: write tcp 188.166.225.170:45400->104.36.82.55:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:16:31.786762817 (60h40m41.509325857s) [seg_pos={0 0}, upload=58175]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:16:32.017758212 (60h40m41.740321252s) [upload=58175]: Done uploading segments... Jul 27 00:16:32.017790574 (60h40m41.740353614s) [upload=58175]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.233138499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...XXX..X.....X......X...XX..XX.X.XX...X...............XX...] duration: 30.794972575s FAIL (2025-07-27 00:16:16 UTC) orbiter warning: context deadline exceeded Duration: 30.794984667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.845425682s FAIL (2025-07-27 00:16:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.845453506s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.XX] duration: 30.263820212s FAIL (2025-07-27 00:16:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.263859808s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X.] duration: 6.166259987s FAIL (2025-07-27 00:15:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16628188s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [....XXX..X.....X......X...XX..XX.X.XX...X...............XX..] duration: 57.462811817s FAIL (2025-07-27 00:15:45 UTC) orbiter warning: context deadline exceeded Duration: 57.46282935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX...] duration: 7.080756075s FAIL (2025-07-27 00:15:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.080785682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.242804576s FAIL (2025-07-27 00:15:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== V5L7L6CSLYBWEFJFBXFLBPPEUEOPIUJA, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.391258468 (60h38m42.113821498s) [noise=true , address=104.36.82.47:4544 , piece_id=QNLFG4NBO2YP7HLTH3G33EL53ZYOLGUEPG4YR3LBSIUS4777CIWA, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.391537471 (60h38m42.114100511s) [noise=true , address=104.36.82.14:4238 , piece_id=YCZRG2AFBJIGVTR7USWPRAOVYCA276MO4LTJMZ2YW5PJ5WDHZSTA, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.391765128 (60h38m42.114328158s) [noise=true , address=64.49.13.138:4488 , piece_id=JNPDUNKGTIWZ6GOYRLDVRDTQYRACF3VHRRHDL67MT5XLHAAZLR5Q, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.391844438 (60h38m42.114407478s) [noise=true , address=79.127.219.38:4009 , piece_id=OXNUS27CY6ZFOJWM4O6AMFCEHDDHDNHK4ZDYBYGGY2LU5S2AI7XQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.392099096 (60h38m42.114662136s) [noise=true , address=64.49.13.137:4502 , piece_id=VIREO4L37NOMOET2G5SVGQ547ZNINSEPF7K2XJNIZS26DPCBIALQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.392352261 (60h38m42.114915301s) [noise=true , address=156.146.43.225:4449 , piece_id=WGVSGOEWWCGXGKCIQGFNSGMVYUFT3NUEUAENPUOJPMOYKSPSUHHQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.392421573 (60h38m42.114984613s) [noise=true , address=104.36.82.55:4223 , piece_id=JLHQHWCGLZNX35AE6SPH52MJNPNRXFU2T7A7XHNODV5XNH4IKI4A, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.392722749 (60h38m42.115285779s) [noise=true , address=104.36.82.53:4147 , piece_id=7TNFTZ7UR725IYM42VYQTE6QHEFKB7IDLR56Q2UOBQPQJ3XBJWOQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393015960 (60h38m42.115579s) [noise=true , address=104.36.82.54:4178 , piece_id=KQ3XVRCTO3EO3J3LPCUHVPYPV4A2PP3PZHMFKDHHDKLCB36PHCLQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393139908 (60h38m42.115702948s) [noise=true , address=79.127.220.101:4041 , piece_id=4RZQU52VPJ4GWJKBPUG6RSJ3DSKGMXCIFPCEXH6U6LVOSS4F3ARA, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393310574 (60h38m42.115873624s) [noise=true , address=79.127.219.48:4074 , piece_id=4H5P7FZFPMJPQJQD5PORJCMVFK3J66MWPPAFWHL7I6XHXS355BIQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393457705 (60h38m42.116020745s) [noise=true , address=104.36.82.52:4117 , piece_id=IFNSITG7ESH5OFU5W6DGPQDYYIBGMFEUIO7IMLS6M34TUE7Q7MAQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393593275 (60h38m42.116156315s) [noise=true , address=79.127.205.233:4629 , piece_id=QQZ2T7USYNPLF2SZJ35KSMYQWI2HHG2DB2RF3YY2U2LIBM4L74VQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393706081 (60h38m42.116269121s) [noise=true , address=79.127.201.215:4348 , piece_id=ZZWADSHB7MTNB3LCRRJPQ56BCYUAG2AOHAM6GACAGGYKDW4X4VHQ, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393919950 (60h38m42.11648299s) [noise=true , address=77.90.31.11:4578 , piece_id=HUGZVPZTRKC2HJFS6FC7QYP3MUH7BBNYMK5S74NPASXMBGFQTI2Q, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.393987278 (60h38m42.116550318s) [noise=true , address=79.127.205.246:4284 , piece_id=V7QMS6ZE4UBGE7KZDCLFS46IDXQ4JHRUYKHI7ODMJHAM5UKDTCTA, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:32.394235964 (60h38m42.116798994s) [noise=true , address=45.91.116.9:4277 , piece_id=NPZFTKXSYOZIY24L7EAL2XUVK357ELI7PRWPTJ7F4ROWN53DNN6Q, seg_pos={0 0}, upload=58161]: Uploading piece... Jul 27 00:14:33.481836882 (60h38m43.204399933s) [upload=58161]: Next returned nil segment Jul 27 00:14:33.481862161 (60h38m43.204425201s) [upload=58161]: Waiting for error group managing segments... Jul 27 00:14:33.843055313 (60h38m43.565618343s) [noise=true , address=79.127.205.246:4284 , piece_id=V7QMS6ZE4UBGE7KZDCLFS46IDXQ4JHRUYKHI7ODMJHAM5UKDTCTA, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:33.874528249 (60h38m43.597091289s) [noise=true , address=79.127.205.233:4629 , piece_id=QQZ2T7USYNPLF2SZJ35KSMYQWI2HHG2DB2RF3YY2U2LIBM4L74VQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:33.913368090 (60h38m43.63593113s) [noise=true , address=79.127.219.48:4074 , piece_id=4H5P7FZFPMJPQJQD5PORJCMVFK3J66MWPPAFWHL7I6XHXS355BIQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:127YpFQAmSFbgY2cmAEwcCPDUhg9AnRHS3SkcafEaQeSGVUwVdj, address:79.127.219.48:4074): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:33.950710306 (60h38m43.673273386s) [noise=true , address=64.49.13.137:4502 , piece_id=VIREO4L37NOMOET2G5SVGQ547ZNINSEPF7K2XJNIZS26DPCBIALQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:34.297339864 (60h38m44.019902904s) [noise=true , address=79.127.219.38:4009 , piece_id=OXNUS27CY6ZFOJWM4O6AMFCEHDDHDNHK4ZDYBYGGY2LU5S2AI7XQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:34.582778546 (60h38m44.305341586s) [noise=true , address=77.90.31.11:4578 , piece_id=HUGZVPZTRKC2HJFS6FC7QYP3MUH7BBNYMK5S74NPASXMBGFQTI2Q, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:36.797503521 (60h38m46.520066561s) [noise=true , address=79.127.201.215:4348 , piece_id=ZZWADSHB7MTNB3LCRRJPQ56BCYUAG2AOHAM6GACAGGYKDW4X4VHQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:37.005524963 (60h38m46.728088003s) [noise=true , address=64.49.13.138:4488 , piece_id=JNPDUNKGTIWZ6GOYRLDVRDTQYRACF3VHRRHDL67MT5XLHAAZLR5Q, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:12mQqzHsTJvR5wHpEGXDixk3NcRuBKxY2cL82eJMHidcxFiPCZb, address:64.49.13.138:4488): protocol: storage node overloaded, request limit: 2000 Jul 27 00:14:37.326666375 (60h38m47.049229415s) [noise=true , address=156.146.43.225:4449 , piece_id=WGVSGOEWWCGXGKCIQGFNSGMVYUFT3NUEUAENPUOJPMOYKSPSUHHQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:42.618557362 (60h38m52.341120402s) [noise=true , address=79.127.220.101:4041 , piece_id=4RZQU52VPJ4GWJKBPUG6RSJ3DSKGMXCIFPCEXH6U6LVOSS4F3ARA, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:14:43.151678609 (60h38m52.874241649s) [noise=true , address=45.91.116.9:4277 , piece_id=NPZFTKXSYOZIY24L7EAL2XUVK357ELI7PRWPTJ7F4ROWN53DNN6Q, seg_pos={0 0}, upload=58161]: Done uploading piece. err: <nil> Jul 27 00:15:31.295552499 (60h39m41.018115549s) [noise=true , address=104.36.82.14:4238 , piece_id=YCZRG2AFBJIGVTR7USWPRAOVYCA276MO4LTJMZ2YW5PJ5WDHZSTA, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:104.36.82.14:4238): protocol: rpc: dial tcp 104.36.82.14:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295613847 (60h39m41.018176887s) [noise=true , address=104.36.82.53:4147 , piece_id=7TNFTZ7UR725IYM42VYQTE6QHEFKB7IDLR56Q2UOBQPQJ3XBJWOQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 188.166.225.170:59748->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295650137 (60h39m41.018213177s) [noise=true , address=104.36.82.54:4178 , piece_id=KQ3XVRCTO3EO3J3LPCUHVPYPV4A2PP3PZHMFKDHHDKLCB36PHCLQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: rpc: dial tcp 104.36.82.54:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295698239 (60h39m41.018261279s) [noise=true , address=104.36.82.55:4223 , piece_id=JLHQHWCGLZNX35AE6SPH52MJNPNRXFU2T7A7XHNODV5XNH4IKI4A, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295754957 (60h39m41.018317997s) [noise=true , address=104.36.82.47:4544 , piece_id=QNLFG4NBO2YP7HLTH3G33EL53ZYOLGUEPG4YR3LBSIUS4777CIWA, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:104.36.82.47:4544): protocol: rpc: dial tcp 104.36.82.47:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295830852 (60h39m41.018393892s) [noise=true , address=104.36.82.52:4117 , piece_id=IFNSITG7ESH5OFU5W6DGPQDYYIBGMFEUIO7IMLS6M34TUE7Q7MAQ, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: rpc: dial tcp 104.36.82.52:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:15:31.295840280 (60h39m41.01840332s) [noise=true , address=77.90.31.21:4891 , piece_id=M4XLCLRJ4IF2JF7YTPSDV5L7L6CSLYBWEFJFBXFLBPPEUEOPIUJA, seg_pos={0 0}, upload=58161]: Done uploading piece. err: ecclient: upload failed (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g, address:77.90.31.21:4891): protocol: read tcp 188.166.225.170:33726->77.90.31.21:4891: use of closed network connection; conn closed: op Jul 27 00:15:31.295874786 (60h39m41.018437826s) [seg_pos={0 0}, upload=58161]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:15:31.536090541 (60h39m41.258653581s) [upload=58161]: Done uploading segments... Jul 27 00:15:31.536128704 (60h39m41.258691744s) [upload=58161]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.242817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....X] duration: 30.001823077s FAIL (2025-07-27 00:15:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001877481s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX..] duration: 7.217335991s FAIL (2025-07-27 00:15:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.217379343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.X] duration: 30.261240024s FAIL (2025-07-27 00:15:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.261283407s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.864476784s FAIL (2025-07-27 00:15:08 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.864496091s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX.] duration: 5.786423553s FAIL (2025-07-27 00:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.786453059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235673409s FAIL (2025-07-27 00:14:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 28416903 (60h37m41.650979943s) [seg_pos={0 0}, upload=58147]: Done begin upload segment. Jul 27 00:13:31.928440568 (60h37m41.651003598s) [noise=true , address=104.36.82.56:4236 , piece_id=SU2VYBUWD7AEGLINFQYY5QNIGOBYM5WV5S7AKTAKVDEJQBHT3GIA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928472249 (60h37m41.65103528s) [noise=true , address=104.36.82.54:4178 , piece_id=YF7XKEKKYSTCCGCONGSPIAKW56ZETJX35KP2YIFUKQJRJ3OPEDGA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928832559 (60h37m41.651395589s) [noise=true , address=45.91.116.14:4206 , piece_id=JCUIKJVZTKHHM2BEUVCNQPSMJQZAY2Z4O256UVYIX2BOM4VPAKSQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928888486 (60h37m41.651451526s) [noise=true , address=156.146.43.225:4448 , piece_id=R3EP7X762SBMUQXNYGBWGHN4YUO3NW5HJPB35ILDNBHM5N36FDPA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929108517 (60h37m41.651671557s) [noise=true , address=77.90.31.13:4643 , piece_id=WBG7EOI56MVA5LSKQIOMFUOTG4Q6TYO5NOHLCUUEWTOF3QTSCJEA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929371089 (60h37m41.651934129s) [noise=true , address=79.127.219.48:4074 , piece_id=3NPXRUXYUN2PSC2VPRAGCZXGNJ6K2VYQMH2KACVFDEQCJLTTDTPQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929644582 (60h37m41.652207622s) [noise=true , address=104.36.82.55:4221 , piece_id=IUFDFPVN3WPSDFGXQ77XFLRYQ3MO6Y4NJPQ5AK5OUV7V73ZWZBYQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930034808 (60h37m41.652597848s) [noise=true , address=104.36.82.52:4085 , piece_id=NQM7ZAJBUA3P2DYREEQTFWBBIRQOFASAF7MWUDSVG24TFEYY3X5A, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930361453 (60h37m41.652924493s) [noise=true , address=79.127.205.235:4685 , piece_id=OVNY3HICKYNJYZ3GW4HWO4PZDWVOXGKPAO6AW5M6FKSMCMXGBFOQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930653773 (60h37m41.653216813s) [noise=true , address=77.90.31.15:4717 , piece_id=BBLZR3D5I5DBKWLPIUGZXLN4CKO2P2YTKAPNJMAXURSNFOV6FHSQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930894213 (60h37m41.653457253s) [noise=true , address=77.90.31.16:4737 , piece_id=KBJHI2NZFQOOJBLIZTOBXZ2YWYHTRYD3XT4R32ZKANSGKZXCCP2Q, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931108022 (60h37m41.653671062s) [noise=true , address=79.127.201.215:4353 , piece_id=U4VO7FBOQVXCF3AKCLZWKRGF3F6DFTTOCWLG2EGWKVXKPZ2AOXAQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931359473 (60h37m41.653922513s) [noise=true , address=79.127.219.38:4004 , piece_id=HCMN2IFXDW3DMTSJWYUWZQKNQFHARZZAFEGH36JBFY6FUAKK7IIQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931576077 (60h37m41.654139117s) [noise=true , address=64.49.13.138:4482 , piece_id=YA2UF5GDSI5RJQVQK4OEIP6MHOEVCDCNVPGATKPXXIDIM3CPTZYA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931788273 (60h37m41.654351314s) [noise=true , address=79.127.205.233:4639 , piece_id=ENQHRZAA2PTHZQGCFT6UMSWN3XDT3AIKCNRYPAC4L2J6XRN3MABQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932060815 (60h37m41.654623855s) [noise=true , address=104.36.82.44:4886 , piece_id=7PJECFYRA2F44D4QT2CKSLYQQQ23EIYQ6GBUVUJWJDJBK5DYVIRQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932312446 (60h37m41.654875486s) [noise=true , address=104.36.82.53:4150 , piece_id=NRJTT22H7ZUNQIZLFRSLLWWQR3S3BANFSYWSYVRC5SIALFX6OJ4A, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932543339 (60h37m41.655106379s) [noise=true , address=77.90.31.10:4525 , piece_id=6TPS42S3CQQHF4N3LZ4345MTJATCT7FBJLPMUOUQQMXGBBZZHOPQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:32.958467822 (60h37m42.681030863s) [noise=true , address=156.146.43.225:4448 , piece_id=R3EP7X762SBMUQXNYGBWGHN4YUO3NW5HJPB35ILDNBHM5N36FDPA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:33.047452184 (60h37m42.770015224s) [upload=58147]: Next returned nil segment Jul 27 00:13:33.047490968 (60h37m42.770054008s) [upload=58147]: Waiting for error group managing segments... Jul 27 00:13:33.330630772 (60h37m43.053193842s) [noise=true , address=79.127.205.233:4639 , piece_id=ENQHRZAA2PTHZQGCFT6UMSWN3XDT3AIKCNRYPAC4L2J6XRN3MABQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.348175059 (60h37m43.070738099s) [noise=true , address=79.127.205.235:4685 , piece_id=OVNY3HICKYNJYZ3GW4HWO4PZDWVOXGKPAO6AW5M6FKSMCMXGBFOQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.369863495 (60h37m43.092426565s) [noise=true , address=64.49.13.138:4482 , piece_id=YA2UF5GDSI5RJQVQK4OEIP6MHOEVCDCNVPGATKPXXIDIM3CPTZYA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.951196995 (60h37m43.673760025s) [noise=true , address=77.90.31.15:4717 , piece_id=BBLZR3D5I5DBKWLPIUGZXLN4CKO2P2YTKAPNJMAXURSNFOV6FHSQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.996608721 (60h37m43.719171751s) [noise=true , address=79.127.219.38:4004 , piece_id=HCMN2IFXDW3DMTSJWYUWZQKNQFHARZZAFEGH36JBFY6FUAKK7IIQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:34.063162086 (60h37m43.785725126s) [noise=true , address=77.90.31.16:4737 , piece_id=KBJHI2NZFQOOJBLIZTOBXZ2YWYHTRYD3XT4R32ZKANSGKZXCCP2Q, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:34.255635539 (60h37m43.978198599s) [noise=true , address=79.127.219.48:4074 , piece_id=3NPXRUXYUN2PSC2VPRAGCZXGNJ6K2VYQMH2KACVFDEQCJLTTDTPQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:35.905447101 (60h37m45.628010131s) [noise=true , address=45.91.116.14:4206 , piece_id=JCUIKJVZTKHHM2BEUVCNQPSMJQZAY2Z4O256UVYIX2BOM4VPAKSQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:47.894801826 (60h37m57.617364866s) [noise=true , address=79.127.201.215:4353 , piece_id=U4VO7FBOQVXCF3AKCLZWKRGF3F6DFTTOCWLG2EGWKVXKPZ2AOXAQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:48.375089258 (60h37m58.097652298s) [noise=true , address=77.90.31.13:4643 , piece_id=WBG7EOI56MVA5LSKQIOMFUOTG4Q6TYO5NOHLCUUEWTOF3QTSCJEA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:58.109318546 (60h38m7.831881576s) [noise=true , address=77.90.31.10:4525 , piece_id=6TPS42S3CQQHF4N3LZ4345MTJATCT7FBJLPMUOUQQMXGBBZZHOPQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:14:30.836823416 (60h38m40.559386466s) [noise=true , address=104.36.82.52:4085 , piece_id=NQM7ZAJBUA3P2DYREEQTFWBBIRQOFASAF7MWUDSVG24TFEYY3X5A, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:37954->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.836916514 (60h38m40.559479575s) [noise=true , address=104.36.82.56:4236 , piece_id=SU2VYBUWD7AEGLINFQYY5QNIGOBYM5WV5S7AKTAKVDEJQBHT3GIA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:51754->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837056552 (60h38m40.559619612s) [noise=true , address=104.36.82.44:4886 , piece_id=7PJECFYRA2F44D4QT2CKSLYQQQ23EIYQ6GBUVUJWJDJBK5DYVIRQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837158728 (60h38m40.559721808s) [noise=true , address=104.36.82.55:4221 , piece_id=IUFDFPVN3WPSDFGXQ77XFLRYQ3MO6Y4NJPQ5AK5OUV7V73ZWZBYQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: write tcp 188.166.225.170:58344->104.36.82.55:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837184107 (60h38m40.559747187s) [noise=true , address=104.36.82.53:4150 , piece_id=NRJTT22H7ZUNQIZLFRSLLWWQR3S3BANFSYWSYVRC5SIALFX6OJ4A, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: write tcp 188.166.225.170:55702->104.36.82.53:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837234262 (60h38m40.559797302s) [noise=true , address=104.36.82.54:4178 , piece_id=YF7XKEKKYSTCCGCONGSPIAKW56ZETJX35KP2YIFUKQJRJ3OPEDGA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: write tcp 188.166.225.170:48602->104.36.82.54:4178: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837267857 (60h38m40.559830927s) [seg_pos={0 0}, upload=58147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:14:31.070880202 (60h38m40.793443242s) [upload=58147]: Done uploading segments... Jul 27 00:14:31.070906612 (60h38m40.793469642s) [upload=58147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.235686033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....] duration: 7.305369763s FAIL (2025-07-27 00:14:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.305406684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.831017177s FAIL (2025-07-27 00:14:26 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.831049789s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX....] duration: 7.252766366s FAIL (2025-07-27 00:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.252984685s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.....XXX..X.....X......X...XX..XX.X.XX...X...............XX.] duration: 29.825325554s FAIL (2025-07-27 00:14:03 UTC) orbiter warning: context deadline exceeded Duration: 29.825338108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX] duration: 30.237561816s FAIL (2025-07-27 00:13:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.237579049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.] duration: 5.931110717s FAIL (2025-07-27 00:13:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.931135334s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.....XXX..X.....X......X...XX..XX.X.XX...X...............XX] duration: 1m0.001316119s FAIL (2025-07-27 00:13:33 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001327992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX] duration: 30.235165588s FAIL (2025-07-27 00:13:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235191346s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233159098s FAIL (2025-07-27 00:13:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== pb.BatchRequestItem_ObjectBegin Jul 27 00:12:31.026903870 (60h36m40.74946691s) [upload=58129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:12:31.478225380 (60h36m41.20078842s) [seg_pos={0 0}, upload=58129]: Begin upload segment... Jul 27 00:12:31.478278161 (60h36m41.200841201s) [seg_pos={0 0}, upload=58129]: Done begin upload segment. Jul 27 00:12:31.478301195 (60h36m41.200864235s) [noise=true , address=79.127.201.215:4350 , piece_id=AP7PIMDUSN6LTBE34B3TKHGWYSNRWCQAKPWLH44N2NHKAKDXOK3A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478354848 (60h36m41.200917888s) [noise=true , address=79.127.220.101:4041 , piece_id=6DVKN65Q7VVWS7GHJ3ZGYD4XQTYT5W6YF47WPXSMJOYSHVU4T35A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478654591 (60h36m41.201217632s) [noise=true , address=104.36.82.47:4534 , piece_id=43HSPMEYXZYDMCRQTWKOJX6SSIOUCWSWCC3PIVBCIEJPUO3GA3XA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478873220 (60h36m41.20143626s) [noise=true , address=104.36.82.46:4484 , piece_id=CRFU7MWRA7YIOMOTX32Q26YROGJN5MRF4IVMX4VQYX25Z2S36UEQ, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479186629 (60h36m41.201749669s) [noise=true , address=77.90.31.10:4531 , piece_id=BJCL4MHMZPPF24ESZRGTR2PFMFMAU3F3W7PJTIP2TSGSZJZASBFA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479239400 (60h36m41.20180244s) [noise=true , address=79.127.205.234:4614 , piece_id=QY5DPWNHPRH6RK3SW3IQQ2BNK2PEOM5D6U3FJAU5E5A3VVUE4B7A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479496552 (60h36m41.202059592s) [noise=true , address=79.127.205.235:4682 , piece_id=T2NH7OPDTYKY5G54TRIG5CP4AXX3PTWWM6OAMTBN6QYC3T2BSU5Q, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479544854 (60h36m41.202107894s) [noise=true , address=79.127.219.47:4048 , piece_id=6MV33SFVBT5QMIRGEL7JVQ3YL4F2URONBGJVJTMPMASGHMMIJMUA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479733475 (60h36m41.202296516s) [noise=true , address=79.127.220.98:4422 , piece_id=DICLWS5NHPUEPN2ZVYVMTN4S654A43QDLTJ2IQWAIY5HJZ4EIX3A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479872942 (60h36m41.202435982s) [noise=true , address=104.36.82.55:4211 , piece_id=F3OBX672E4BWQ2YU7NOR4U3QY4EYW5OOYQLUMLT4UPDDNRJ4W7IA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479971742 (60h36m41.202534782s) [noise=true , address=79.127.219.38:4004 , piece_id=CA4FYHIK3F5YQ5V5KX7IV3QBAYVBHGNZAGD5TDKM5V54IQGWNH7A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480208815 (60h36m41.202771855s) [noise=true , address=79.127.205.233:4624 , piece_id=JSUZMMEKY2RUVRJ2XCOICYE3OK33IQP33NBOCG5Z7ULBYFLF7TKQ, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480278288 (60h36m41.202841328s) [noise=true , address=64.49.13.138:4483 , piece_id=QFIH742V62JXFP5KLTLNA32PAFW5QVJF5TGZNVC7OBYJ7HLLXITA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480506044 (60h36m41.203069084s) [noise=true , address=104.36.82.53:4149 , piece_id=Q4S7UJRDCIS3473MSSXFPAR6WJMF76OWDD7SMA5MGUKJTOEXM36Q, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480705956 (60h36m41.203268986s) [noise=true , address=45.91.116.9:4277 , piece_id=YLHBIM36PKVKK2FKBYNE4QTXHIGQRXHNFSDUZSPP74AJUWZEUAJA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480910428 (60h36m41.203473458s) [noise=true , address=104.36.82.52:4109 , piece_id=62HBDBLDIFA4NIIJK6FKSYTHLVYHHXE7RTW4VZFQM73NE53BU7DA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.481107134 (60h36m41.203670174s) [noise=true , address=45.91.116.14:4206 , piece_id=2MFVCUVABQSY7B6SQK3NKFR26UGQZKZYSTTGTUZVX4HNO35EDTDA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.481308018 (60h36m41.203871068s) [noise=true , address=79.127.220.101:4048 , piece_id=EVV7G4MS43G6RTQZMTB4TZBWPN52HJB5CFZHZXECYNAMA3EGRG6A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:32.575385570 (60h36m42.29794861s) [upload=58129]: Next returned nil segment Jul 27 00:12:32.575402632 (60h36m42.297965672s) [upload=58129]: Waiting for error group managing segments... Jul 27 00:12:32.877416878 (60h36m42.599979918s) [noise=true , address=79.127.205.233:4624 , piece_id=JSUZMMEKY2RUVRJ2XCOICYE3OK33IQP33NBOCG5Z7ULBYFLF7TKQ, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:32.882944154 (60h36m42.605507194s) [noise=true , address=79.127.205.234:4614 , piece_id=QY5DPWNHPRH6RK3SW3IQQ2BNK2PEOM5D6U3FJAU5E5A3VVUE4B7A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:32.883372023 (60h36m42.605935063s) [noise=true , address=79.127.205.235:4682 , piece_id=T2NH7OPDTYKY5G54TRIG5CP4AXX3PTWWM6OAMTBN6QYC3T2BSU5Q, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:33.354169718 (60h36m43.076732758s) [noise=true , address=79.127.219.38:4004 , piece_id=CA4FYHIK3F5YQ5V5KX7IV3QBAYVBHGNZAGD5TDKM5V54IQGWNH7A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:33.390453408 (60h36m43.113016438s) [noise=true , address=79.127.201.215:4350 , piece_id=AP7PIMDUSN6LTBE34B3TKHGWYSNRWCQAKPWLH44N2NHKAKDXOK3A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.208209173 (60h36m44.930772214s) [noise=true , address=77.90.31.10:4531 , piece_id=BJCL4MHMZPPF24ESZRGTR2PFMFMAU3F3W7PJTIP2TSGSZJZASBFA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.560171254 (60h36m45.282734444s) [noise=true , address=79.127.219.47:4048 , piece_id=6MV33SFVBT5QMIRGEL7JVQ3YL4F2URONBGJVJTMPMASGHMMIJMUA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.817925048 (60h36m45.540488088s) [noise=true , address=79.127.220.101:4048 , piece_id=EVV7G4MS43G6RTQZMTB4TZBWPN52HJB5CFZHZXECYNAMA3EGRG6A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:36.023331778 (60h36m45.745894808s) [noise=true , address=64.49.13.138:4483 , piece_id=QFIH742V62JXFP5KLTLNA32PAFW5QVJF5TGZNVC7OBYJ7HLLXITA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:36.431969173 (60h36m46.154532213s) [noise=true , address=45.91.116.9:4277 , piece_id=YLHBIM36PKVKK2FKBYNE4QTXHIGQRXHNFSDUZSPP74AJUWZEUAJA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:12:37.019430645 (60h36m46.741993675s) [noise=true , address=79.127.220.98:4422 , piece_id=DICLWS5NHPUEPN2ZVYVMTN4S654A43QDLTJ2IQWAIY5HJZ4EIX3A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:37.140244517 (60h36m46.862807547s) [noise=true , address=79.127.220.101:4041 , piece_id=6DVKN65Q7VVWS7GHJ3ZGYD4XQTYT5W6YF47WPXSMJOYSHVU4T35A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:37.869550775 (60h36m47.592113815s) [noise=true , address=45.91.116.14:4206 , piece_id=2MFVCUVABQSY7B6SQK3NKFR26UGQZKZYSTTGTUZVX4HNO35EDTDA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:13:30.379652793 (60h37m40.102215843s) [noise=true , address=104.36.82.46:4484 , piece_id=CRFU7MWRA7YIOMOTX32Q26YROGJN5MRF4IVMX4VQYX25Z2S36UEQ, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 188.166.225.170:56326->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379718449 (60h37m40.102281489s) [noise=true , address=104.36.82.55:4211 , piece_id=F3OBX672E4BWQ2YU7NOR4U3QY4EYW5OOYQLUMLT4UPDDNRJ4W7IA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379795475 (60h37m40.102358516s) [noise=true , address=104.36.82.47:4534 , piece_id=43HSPMEYXZYDMCRQTWKOJX6SSIOUCWSWCC3PIVBCIEJPUO3GA3XA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: write tcp 188.166.225.170:52358->104.36.82.47:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379802490 (60h37m40.10236554s) [noise=true , address=104.36.82.53:4149 , piece_id=Q4S7UJRDCIS3473MSSXFPAR6WJMF76OWDD7SMA5MGUKJTOEXM36Q, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: write tcp 188.166.225.170:37754->104.36.82.53:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379870640 (60h37m40.10243369s) [noise=true , address=104.36.82.52:4109 , piece_id=62HBDBLDIFA4NIIJK6FKSYTHLVYHHXE7RTW4VZFQM73NE53BU7DA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 188.166.225.170:41724->104.36.82.52:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379910366 (60h37m40.102473406s) [seg_pos={0 0}, upload=58129]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:13:30.611392384 (60h37m40.333955424s) [upload=58129]: Done uploading segments... Jul 27 00:13:30.611412573 (60h37m40.333975613s) [upload=58129]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.233169979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.991619004s FAIL (2025-07-27 00:13:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.991647769s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX...] duration: 6.20457161s FAIL (2025-07-27 00:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.204584865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..X] duration: 30.232647615s FAIL (2025-07-27 00:12:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== timal: 13 errs: <nil> Jul 27 00:11:50.257632003 (60h35m59.980195043s) [upload=58119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:11:50.257636792 (60h35m59.980199832s) [upload=58119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:11:50.547920805 (60h36m0.270483835s) [upload=58119]: Done uploading segments... Jul 27 00:11:50.547935042 (60h36m0.270498072s) [upload=58119]: Upload finished. err: <nil> Jul 27 00:12:15.130189924 (60h36m24.852752964s) [upload=58125]: Starting upload Jul 27 00:12:15.130249278 (60h36m24.852812318s) [upload=58125]: Done starting upload Jul 27 00:12:15.130350221 (60h36m24.852913261s) [upload=58125]: Uploading segments... Jul 27 00:12:15.130485049 (60h36m24.853048089s) [upload=58125]: Got next segment. Inline: false Jul 27 00:12:15.130517591 (60h36m24.853080631s) [upload=58125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:12:15.130522711 (60h36m24.853085751s) [upload=58125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:12:15.364289469 (60h36m25.086852509s) [seg_pos={0 0}, upload=58125]: Begin upload segment... Jul 27 00:12:15.364335027 (60h36m25.086898067s) [seg_pos={0 0}, upload=58125]: Done begin upload segment. Jul 27 00:12:15.364358562 (60h36m25.086921602s) [noise=true , address=77.90.31.18:4805 , piece_id=J3J3BGYQ3P56A5XHXTKV76LNTDYH7TVICAWH347OTNCBJGQWM32A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364383348 (60h36m25.086946378s) [noise=true , address=77.90.31.11:4555 , piece_id=WGHTIXXMTXXHPWHOMD6LBBLM6FH7ZVKD6EVFEVKW4SR36VTUB4UA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364686328 (60h36m25.087249358s) [noise=true , address=45.91.116.6:4975 , piece_id=NONSFAX777AESHHYK5GMZQSWACODPXTFDE5NYVWINAFKP2GXBNMA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364796078 (60h36m25.087359118s) [noise=true , address=104.36.82.5:4840 , piece_id=6NZXAQI2PHYYDGMB2SNK376D2ZPAHGAMDUC2L22IV5OEHILT4M2A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364930625 (60h36m25.087493656s) [noise=true , address=45.91.116.9:4263 , piece_id=4EXFMBDDM6Q3NR2S3OZZ2XZVXA23MNFCWYYR466GLWRD5KJGAWCA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365169252 (60h36m25.087732292s) [noise=true , address=77.90.31.10:4520 , piece_id=TFGJVDUPYPHIAUPQ3ANCDWFNRJPGFNJ74O7YCA4WCLRZYJUIBAYQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365396016 (60h36m25.087959056s) [noise=true , address=45.91.116.10:4081 , piece_id=BGYHTRQB7OVMGD3WKD7SFT6E75A47JKKDEWOT7GI2VPFOCZ6JQLQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365604796 (60h36m25.088167836s) [noise=true , address=104.36.82.5:4838 , piece_id=WRYCJUJ3X7ZSGW5RJ6EZBLCCV5XOEWO6XXMDPRCFBUNUABYHSAAA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365864532 (60h36m25.088427572s) [noise=true , address=77.90.31.15:4698 , piece_id=LO6AHA6D2JVB6RVNSPOO3DOIGGSK6JC3WGC2USRTP7J5WM5WGXLQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366087559 (60h36m25.088650619s) [noise=true , address=104.36.82.8:4972 , piece_id=K44LZEJKLHTSLVRZJRCOYGPDAZHH4MQBY7IVZZVY2DY4GVVBXAGA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366300998 (60h36m25.088864038s) [noise=true , address=45.91.116.6:4974 , piece_id=4TIJOOEAJ3Q7474QI75O56GOCW3XTTI75BS3N72V2ILWPIJTUUHQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366504057 (60h36m25.089067097s) [noise=true , address=45.91.116.4:4940 , piece_id=SZCTGPH42EQB3ZOK4D5R667Q64S3WIWXFUCSWLZMEEHRCYFQJH6A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366686165 (60h36m25.089249205s) [noise=true , address=45.91.116.14:4191 , piece_id=LMQLC4GCAVBCBVHCHJYDIWF7CBSEXOVN5WF3HJAJP7QPON6QMNKQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366874625 (60h36m25.089437665s) [noise=true , address=104.36.82.14:4222 , piece_id=EGTCKWAJISTAI3KJLN3G3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.367061613 (60h36m25.089624653s) [noise=true , address=45.91.116.10:4083 , piece_id=BC5POQB34FGQ2S2YDGAXBPYOXWXH6JLQYYGQ5JOFIKC4ICUIWR2Q, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.367246868 (60h36m25.089809908s) [noise=true , address=104.36.82.12:4149 , piece_id=YAKK7MXZG3YCVGOAX5L7VWVFHGGSHX6YABNRFLNAMX62QTW426XQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:17.118387739 (60h36m26.840950779s) [upload=58125]: Next returned nil segment Jul 27 00:12:17.118424640 (60h36m26.84098768s) [upload=58125]: Waiting for error group managing segments... Jul 27 00:12:17.395594322 (60h36m27.118157362s) [noise=true , address=77.90.31.15:4698 , piece_id=LO6AHA6D2JVB6RVNSPOO3DOIGGSK6JC3WGC2USRTP7J5WM5WGXLQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.403140179 (60h36m27.125703219s) [noise=true , address=45.91.116.10:4081 , piece_id=BGYHTRQB7OVMGD3WKD7SFT6E75A47JKKDEWOT7GI2VPFOCZ6JQLQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.456923140 (60h36m27.17948618s) [noise=true , address=45.91.116.10:4083 , piece_id=BC5POQB34FGQ2S2YDGAXBPYOXWXH6JLQYYGQ5JOFIKC4ICUIWR2Q, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.470870104 (60h36m27.193433134s) [noise=true , address=45.91.116.6:4974 , piece_id=4TIJOOEAJ3Q7474QI75O56GOCW3XTTI75BS3N72V2ILWPIJTUUHQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.485776194 (60h36m27.208339224s) [noise=true , address=45.91.116.6:4975 , piece_id=NONSFAX777AESHHYK5GMZQSWACODPXTFDE5NYVWINAFKP2GXBNMA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.529157194 (60h36m27.251720234s) [noise=true , address=45.91.116.4:4940 , piece_id=SZCTGPH42EQB3ZOK4D5R667Q64S3WIWXFUCSWLZMEEHRCYFQJH6A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.566592398 (60h36m27.289155438s) [noise=true , address=77.90.31.11:4555 , piece_id=WGHTIXXMTXXHPWHOMD6LBBLM6FH7ZVKD6EVFEVKW4SR36VTUB4UA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.581888173 (60h36m27.304451213s) [noise=true , address=45.91.116.9:4263 , piece_id=4EXFMBDDM6Q3NR2S3OZZ2XZVXA23MNFCWYYR466GLWRD5KJGAWCA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.596212489 (60h36m27.318775529s) [noise=true , address=45.91.116.14:4191 , piece_id=LMQLC4GCAVBCBVHCHJYDIWF7CBSEXOVN5WF3HJAJP7QPON6QMNKQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:18.935772373 (60h36m28.658335413s) [noise=true , address=77.90.31.10:4520 , piece_id=TFGJVDUPYPHIAUPQ3ANCDWFNRJPGFNJ74O7YCA4WCLRZYJUIBAYQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:44.482920968 (60h36m54.205484008s) [noise=true , address=104.36.82.5:4840 , piece_id=6NZXAQI2PHYYDGMB2SNK376D2ZPAHGAMDUC2L22IV5OEHILT4M2A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.482972567 (60h36m54.205535597s) [noise=true , address=104.36.82.5:4838 , piece_id=WRYCJUJ3X7ZSGW5RJ6EZBLCCV5XOEWO6XXMDPRCFBUNUABYHSAAA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:44666->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483121992 (60h36m54.205685022s) [noise=true , address=77.90.31.18:4805 , piece_id=J3J3BGYQ3P56A5XHXTKV76LNTDYH7TVICAWH347OTNCBJGQWM32A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483161168 (60h36m54.205724208s) [noise=true , address=104.36.82.12:4149 , piece_id=YAKK7MXZG3YCVGOAX5L7VWVFHGGSHX6YABNRFLNAMX62QTW426XQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:49192->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483192738 (60h36m54.205755808s) [noise=true , address=104.36.82.8:4972 , piece_id=K44LZEJKLHTSLVRZJRCOYGPDAZHH4MQBY7IVZZVY2DY4GVVBXAGA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483211874 (60h36m54.205774904s) [noise=true , address=104.36.82.14:4222 , piece_id=EGTCKWAJISTAI3KJLN3G3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23267066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...X] duration: 30.235676933s FAIL (2025-07-27 00:12:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.235719543s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.874476376s FAIL (2025-07-27 00:12:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874508218s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XX.....XXX..X.....X......X...XX..XX.X.XX...X...............X] duration: 1m0.000491142s FAIL (2025-07-27 00:12:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000520688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23181715s FAIL (2025-07-27 00:12:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece... Jul 27 00:11:31.017033996 (60h35m40.739597036s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017246372 (60h35m40.739809412s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017307839 (60h35m40.739870879s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017570702 (60h35m40.740133733s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017637410 (60h35m40.74020045s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017870216 (60h35m40.740433256s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017953896 (60h35m40.740516936s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018152706 (60h35m40.740715746s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018210096 (60h35m40.740773136s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018371675 (60h35m40.740934705s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018440778 (60h35m40.741003818s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018729870 (60h35m40.74129291s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018981191 (60h35m40.741544221s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019211282 (60h35m40.741774322s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019456681 (60h35m40.742019721s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019673756 (60h35m40.742236796s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019901743 (60h35m40.742464783s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:32.116889070 (60h35m41.83945211s) [upload=58114]: Next returned nil segment Jul 27 00:11:32.116907094 (60h35m41.839470135s) [upload=58114]: Waiting for error group managing segments... Jul 27 00:11:32.400438007 (60h35m42.123001047s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.419754955 (60h35m42.142317995s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.454796929 (60h35m42.17735997s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.508361473 (60h35m42.230924514s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:33.092872603 (60h35m42.815435643s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:34.345847555 (60h35m44.068410595s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.620976689 (60h35m46.343539729s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.896156473 (60h35m46.618719513s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:40.115868474 (60h35m49.838431514s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:49.826606684 (60h35m59.549169724s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:12:29.931183938 (60h36m39.653746978s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: write tcp 188.166.225.170:40114->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931219377 (60h36m39.653782487s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: write tcp 188.166.225.170:48958->104.36.82.51:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931255435 (60h36m39.653818476s) [noise=true , address=104.36.82.54:4173 , piece_id=DN3POBT5YQ5G27ZN2TTHITQD47ZH5M5CK5X5CRWIVP5Y7QKOS2HA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: write tcp 188.166.225.170:55092->104.36.82.54:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931289260 (60h36m39.6538523s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 188.166.225.170:33636->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931319177 (60h36m39.653882207s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: write tcp 188.166.225.170:38452->104.36.82.15:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931322904 (60h36m39.653886004s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931339937 (60h36m39.653903017s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931349455 (60h36m39.653912485s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:4461): protocol: rpc: dial tcp 104.36.82.20:4461: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931371597 (60h36m39.653934648s) [seg_pos={0 0}, upload=58114]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:12:30.160452502 (60h36m39.883015582s) [upload=58114]: Done uploading segments... Jul 27 00:12:30.160484543 (60h36m39.883047583s) [upload=58114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231844842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX..] duration: 6.231432698s FAIL (2025-07-27 00:12:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231454349s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.] duration: 14.163498248s FAIL (2025-07-27 00:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.163549446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..] duration: 7.096051247s FAIL (2025-07-27 00:11:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.096076656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...] duration: 7.473576427s FAIL (2025-07-27 00:11:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.473602526s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX] duration: 30.000885718s FAIL (2025-07-27 00:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00090808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.953156002s FAIL (2025-07-27 00:11:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.953188435s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230953081s FAIL (2025-07-27 00:11:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.960984242 (60h34m39.683547312s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961282062 (60h34m39.683845102s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961544423 (60h34m39.684107464s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961578839 (60h34m39.684141879s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961829630 (60h34m39.68439267s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961851902 (60h34m39.684414932s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962092733 (60h34m39.684655773s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962165031 (60h34m39.684728081s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962360525 (60h34m39.684923565s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962406674 (60h34m39.684969704s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962613269 (60h34m39.685176299s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962675848 (60h34m39.685238888s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962830334 (60h34m39.685393374s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962928943 (60h34m39.685491983s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963106011 (60h34m39.685669052s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963146589 (60h34m39.685709629s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.965030644 (60h34m39.687593684s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:31.057425385 (60h34m40.779988425s) [upload=58101]: Next returned nil segment Jul 27 00:10:31.057442587 (60h34m40.780005617s) [upload=58101]: Waiting for error group managing segments... Jul 27 00:10:31.358062366 (60h34m41.080625406s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.365146409 (60h34m41.087709439s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.394481485 (60h34m41.117044525s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.451086155 (60h34m41.173649195s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, address:45.91.116.8:4062): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:31.944066471 (60h34m41.666629501s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:32.067400052 (60h34m41.789963092s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:34.098541683 (60h34m43.821104723s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.030914207 (60h34m44.753477247s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.422624445 (60h34m45.145187485s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.728360292 (60h34m45.450923322s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:38.214886788 (60h34m47.937449888s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:11:28.856710878 (60h35m38.579273908s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856757427 (60h35m38.579320457s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856831208 (60h35m38.579394238s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: write tcp 188.166.225.170:50928->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856960185 (60h35m38.579523225s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857005111 (60h35m38.579568191s) [noise=true , address=104.36.82.54:4166 , piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 188.166.225.170:48336->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857048403 (60h35m38.579611443s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:104.36.82.38:4689): protocol: write tcp 188.166.225.170:52234->104.36.82.38:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857114791 (60h35m38.579677831s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:44362->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857138366 (60h35m38.579701406s) [seg_pos={0 0}, upload=58101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:11:29.085947751 (60h35m38.808510791s) [upload=58101]: Done uploading segments... Jul 27 00:11:29.085980072 (60h35m38.808543133s) [upload=58101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230977779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXX.....XXX..X.....X......X...XX..XX.X.XX...X...............] duration: 18.044760268s FAIL (2025-07-27 00:11:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.044788652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.874451351s FAIL (2025-07-27 00:10:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874501938s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)