Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...X.X.X..X.......X.XX.X........XX.........................X] duration:
30.001121378s
FAIL (2025-09-22 05:04:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001173427s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X...X......X.X....X.....X....X.....X.X....XX............XXX] duration:
30.250304743s
FAIL (2025-09-22 05:04:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Sep 22 05:04:01.402889922 (375h35m37.442463052s) [upload=330134]: Starting upload
Sep 22 05:04:01.403012014 (375h35m37.442585144s) [upload=330134]: Done starting upload
Sep 22 05:04:01.403141852 (375h35m37.442714982s) [upload=330134]: Uploading segments...
Sep 22 05:04:01.403362962 (375h35m37.442936112s) [upload=330134]: Got next segment. Inline: false
Sep 22 05:04:01.403422636 (375h35m37.442995766s) [upload=330134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:04:01.403454016 (375h35m37.443027146s) [upload=330134]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:04:01.656162313 (375h35m37.695735453s) [seg_pos={0 0}, upload=330134]: Begin upload segment...
Sep 22 05:04:01.656223539 (375h35m37.695796669s) [seg_pos={0 0}, upload=330134]: Done begin upload segment.
Sep 22 05:04:01.656261551 (375h35m37.695834681s) [noise=true , address=145.79.227.2:4114 , piece_id=7S3XWX2MHRDTWYWDYL3QJIHRFCEK3AS5ZZZCTAY2KKLL5GHC6CSA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.656496729 (375h35m37.696069859s) [noise=true , address=145.79.227.4:4186 , piece_id=XMIRDVNZB6WKBQMEW3XPBS5H7UXPVHNCTZTCX26GSLQYOPAZICHQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.656972335 (375h35m37.696545475s) [noise=true , address=45.91.116.5:4298 , piece_id=CHG5OJCROOUGGK32XC4O36KRKCGHNIPMLLOJFZNDUEVTKTOTCUXA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.657196201 (375h35m37.696769331s) [noise=true , address=45.91.116.7:4009 , piece_id=3SNZQ4PKPJW6E4WBQFTT25YWMMINAQH25KVZL3OSXE2ERYLZOLIQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.657397134 (375h35m37.696970264s) [noise=true , address=77.90.31.13:4626 , piece_id=VPCBAQ35FYUKZRJXJZA7DHWDQSR2OLWKCCYVVW7KNAH6CT3NTINA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.657727844 (375h35m37.697300974s) [noise=true , address=45.91.116.14:4190 , piece_id=2ID7365DAZ33FW6M4P5ZR6J7MTUOR7RJEA2H4NYRONNVFBVIYEHQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.657871868 (375h35m37.697444998s) [noise=true , address=145.79.227.5:4223 , piece_id=WUFGVQUZTFSRN3YRMV5XYTNGPC3IABQ2OJNGUQ7EKA6YVNKTXINQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.658293130 (375h35m37.69786627s) [noise=true , address=145.79.227.12:4767 , piece_id=PBPF7GR3HXIXANSMN5UH74BDEC6KYDW6KOJ5IAVWVK3P7RK7WQTA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.658346401 (375h35m37.697919531s) [noise=true , address=145.79.227.14:4839 , piece_id=RTVG2KGLZRXROCOTAPA4PMBZKM76THHTRUDNOUJXU2BTEMTC23VA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.658692620 (375h35m37.69826575s) [noise=true , address=77.90.31.11:4553 , piece_id=Q36T6CNAD5TA6M3LTR43OTUGQDL6M5WCD243K3234M3UNFFAGHKA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.658906798 (375h35m37.698479958s) [noise=true , address=45.91.116.8:4045 , piece_id=U4BU6OBLQ63O7W22R5M4WTNX5FTSWYP3LL3OW4IMABR57V3JFNKQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.659116849 (375h35m37.698689979s) [noise=true , address=45.91.116.12:4225 , piece_id=ZPUXWFHTHS7W4KOZ337BP6SGS4I4HCQV4DZOP63WZBUE523VZ6EQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.659539243 (375h35m37.699112373s) [noise=true , address=77.90.31.18:4807 , piece_id=SVB6I4GC5U5ANJ3J3G3SDSH2PTJ6WT7PLWA5BVNQRFQYJGJW7GAA, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.659556095 (375h35m37.699129225s) [noise=true , address=145.79.227.11:4729 , piece_id=47TTLFNLELR3QMP5HF7SS7WAADCJMJDSOFQOURBEW2VKJULSV6XQ, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.659960275 (375h35m37.699533405s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZW7K5FPWS65B26JZ2YQF4RUXO4MBZGWXLYQFGO367LJZVNNJN7Q, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:01.660106062 (375h35m37.699679192s) [noise=true , address=77.90.31.11:4555 , piece_id=WOLGAIV52ECTOKFD7SBEC6A6R2RY3KP63HG77SEZVEDTKHLFDB4Q, seg_pos={0 0}, upload=330134]: Uploading piece...
Sep 22 05:04:03.359813993 (375h35m39.399387123s) [upload=330134]: Next returned nil segment
Sep 22 05:04:03.359835994 (375h35m39.399409115s) [upload=330134]: Waiting for error group managing segments...
Sep 22 05:04:03.756513046 (375h35m39.796086176s) [noise=true , address=45.91.116.14:4190 , piece_id=2ID7365DAZ33FW6M4P5ZR6J7MTUOR7RJEA2H4NYRONNVFBVIYEHQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.762894398 (375h35m39.802467528s) [noise=true , address=45.91.116.7:4009 , piece_id=3SNZQ4PKPJW6E4WBQFTT25YWMMINAQH25KVZL3OSXE2ERYLZOLIQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.771638909 (375h35m39.811212039s) [noise=true , address=77.90.31.18:4807 , piece_id=SVB6I4GC5U5ANJ3J3G3SDSH2PTJ6WT7PLWA5BVNQRFQYJGJW7GAA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.789569431 (375h35m39.829142561s) [noise=true , address=45.91.116.8:4045 , piece_id=U4BU6OBLQ63O7W22R5M4WTNX5FTSWYP3LL3OW4IMABR57V3JFNKQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.806082646 (375h35m39.845655786s) [noise=true , address=45.91.116.5:4298 , piece_id=CHG5OJCROOUGGK32XC4O36KRKCGHNIPMLLOJFZNDUEVTKTOTCUXA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.826714774 (375h35m39.866287904s) [noise=true , address=77.90.31.11:4553 , piece_id=Q36T6CNAD5TA6M3LTR43OTUGQDL6M5WCD243K3234M3UNFFAGHKA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.829483526 (375h35m39.869056646s) [noise=true , address=77.90.31.11:4555 , piece_id=WOLGAIV52ECTOKFD7SBEC6A6R2RY3KP63HG77SEZVEDTKHLFDB4Q, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.891717121 (375h35m39.931290251s) [noise=true , address=45.91.116.12:4225 , piece_id=ZPUXWFHTHS7W4KOZ337BP6SGS4I4HCQV4DZOP63WZBUE523VZ6EQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.955144129 (375h35m39.994717259s) [noise=true , address=77.90.31.13:4626 , piece_id=VPCBAQ35FYUKZRJXJZA7DHWDQSR2OLWKCCYVVW7KNAH6CT3NTINA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: <nil>
Sep 22 05:04:03.986483355 (375h35m40.026056485s) [noise=true , address=145.79.227.4:4186 , piece_id=XMIRDVNZB6WKBQMEW3XPBS5H7UXPVHNCTZTCX26GSLQYOPAZICHQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: connect: no route to host; conn closed: op
Sep 22 05:04:30.690767394 (375h36m6.730340524s) [noise=true , address=145.79.227.14:4839 , piece_id=RTVG2KGLZRXROCOTAPA4PMBZKM76THHTRUDNOUJXU2BTEMTC23VA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:30.691111659 (375h36m6.730684789s) [noise=true , address=145.79.227.5:4223 , piece_id=WUFGVQUZTFSRN3YRMV5XYTNGPC3IABQ2OJNGUQ7EKA6YVNKTXINQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: rpc: dial tcp 145.79.227.5:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:30.691180611 (375h36m6.730753741s) [noise=true , address=145.79.227.13:4804 , piece_id=3ZW7K5FPWS65B26JZ2YQF4RUXO4MBZGWXLYQFGO367LJZVNNJN7Q, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; context deadline exceeded
Sep 22 05:04:30.691355804 (375h36m6.730928934s) [noise=true , address=145.79.227.12:4767 , piece_id=PBPF7GR3HXIXANSMN5UH74BDEC6KYDW6KOJ5IAVWVK3P7RK7WQTA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:30.691463699 (375h36m6.73103682s) [noise=true , address=145.79.227.2:4114 , piece_id=7S3XWX2MHRDTWYWDYL3QJIHRFCEK3AS5ZZZCTAY2KKLL5GHC6CSA, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:30.691496201 (375h36m6.731069331s) [noise=true , address=145.79.227.11:4729 , piece_id=47TTLFNLELR3QMP5HF7SS7WAADCJMJDSOFQOURBEW2VKJULSV6XQ, seg_pos={0 0}, upload=330134]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:30.691530767 (375h36m6.731103897s) [seg_pos={0 0}, upload=330134]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Sep 22 05:04:30.937590835 (375h36m6.977163955s) [upload=330134]: Done uploading segments...
Sep 22 05:04:30.937649617 (375h36m6.977222737s) [upload=330134]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.250333609s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.189682971s
FAIL (2025-09-22 05:04:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
*pb.BatchRequestItem_ObjectBegin
Sep 22 05:03:16.206590601 (375h34m52.246163731s) [upload=330124]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:03:16.705371709 (375h34m52.74494485s) [seg_pos={0 0}, upload=330124]: Begin upload segment...
Sep 22 05:03:16.705448145 (375h34m52.745021275s) [seg_pos={0 0}, upload=330124]: Done begin upload segment.
Sep 22 05:03:16.705476790 (375h34m52.74504992s) [noise=true , address=145.79.227.49:4342 , piece_id=3EYT37ETP6Q74YRTJRNJ4JXGD2EVHOSUEWV4KG5XY3FTYYQSS3CQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.705488011 (375h34m52.745061141s) [noise=true , address=145.79.227.13:4835 , piece_id=EXPL7JZEQJBAVZ7R7H4LZ2XGQ2FEILEXKNG3CQCVIM73IE7KRUNQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706086350 (375h34m52.74565948s) [noise=true , address=145.79.227.26:4653 , piece_id=JNAYIWG2T6XR5SPVQT6AKUYCORCKAL7VI62NPTOZDGA2GLMAGCFA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706199506 (375h34m52.745772626s) [noise=true , address=145.79.227.12:4791 , piece_id=FOABF7GEXKWDFSNMX72VROVITWLPI4SA3UKVEWZH64I5FLMCJDHQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706548150 (375h34m52.74612128s) [noise=true , address=145.79.227.43:4166 , piece_id=KY3MDW27JBIPAFCCVWI4DJ4NML45C3NRYQSUMRGRWQB2VQ2NFLAQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707075163 (375h34m52.746648293s) [noise=true , address=145.79.227.40:4068 , piece_id=3JTAFMRWTL55NS6QX73NB63MUKOSEAXTMAELJJBMV2G3JVNE2LQA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707425580 (375h34m52.74699871s) [noise=true , address=145.79.227.24:4604 , piece_id=PAMMKTED4RMO432ZFCSCZI4DS5UFC6SROO3BZUKBGUXCXLJBWK7Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707776068 (375h34m52.747349188s) [noise=true , address=145.79.227.3:4162 , piece_id=Y4ZNYFYUV4VSWYGP3ADJUA43P3XOWT3WCGP2A7H5W5PAZWTL5Z7Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.708123609 (375h34m52.74769674s) [noise=true , address=145.79.227.45:4231 , piece_id=SNVIZK5SFCJ7M66ZW7OUHYPN7X72UKCW6WAARK2IIBJ77HMLMPRQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.708741957 (375h34m52.748315087s) [noise=true , address=77.90.31.13:4653 , piece_id=WUJNKDL6GVG26N4ALSNIQ6TEWEPEUTUMT6GP4ZXATBCGH24MHAXQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709385462 (375h34m52.748958602s) [noise=true , address=145.79.227.23:4578 , piece_id=BVMH3AL2WSIGAXNODTTVEMC3HSC5YL73V6PRMKOANCO7EMUNVUBA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709465565 (375h34m52.749038695s) [noise=true , address=145.79.227.15:4878 , piece_id=SYE46YDTLQQRC2Q2CFCMHG7PDNDK5KIK6PAL7NFEFTJ5FVLGWMOQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709863723 (375h34m52.749436853s) [noise=true , address=77.90.31.11:4566 , piece_id=UC2O2R5VQNGBGSSRMK2FOBTVGW3PQB37654YPBRKRMPD3MNM3NVA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709882649 (375h34m52.74945578s) [noise=true , address=77.90.31.19:4862 , piece_id=XF2X3JOPDY3F56CBLS7NSCMXQDFKFFFEH7FBOFGR5XOWVDTDCWYA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.710501858 (375h34m52.750074988s) [noise=true , address=145.79.227.48:4334 , piece_id=AOGK2ALI4HDA5I624I7ASBBSHA7W6RJJ3XSSHX5IRU2GJWBXRS7A, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.710978395 (375h34m52.750551526s) [noise=true , address=145.79.227.4:4201 , piece_id=PDFMVMTXQ6TYOFLYRR4R34E7EUFI3S24NFW5SARZWTVBB2BMKO6Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.711020215 (375h34m52.750593345s) [noise=true , address=145.79.227.33:4898 , piece_id=5I4XYNRZ2NJI3KNJL2INORWDUTLMDR5NUGWAWAOHQRVN6R4WPDCA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.711453480 (375h34m52.751026611s) [noise=true , address=145.79.227.18:4034 , piece_id=HLWHBRLTVVDMAZMXCTXPPLQYHQYSWENJPXZUDQYH2SKG27EYN75Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:17.877964509 (375h34m53.917537639s) [noise=true , address=145.79.227.4:4201 , piece_id=PDFMVMTXQ6TYOFLYRR4R34E7EUFI3S24NFW5SARZWTVBB2BMKO6Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:145.79.227.4:4201): protocol: rpc: dial tcp 145.79.227.4:4201: connect: no route to host; conn closed: op
Sep 22 05:03:18.187226354 (375h34m54.226799484s) [upload=330124]: Next returned nil segment
Sep 22 05:03:18.187247254 (375h34m54.226820384s) [upload=330124]: Waiting for error group managing segments...
Sep 22 05:03:18.586189148 (375h34m54.625762268s) [noise=true , address=145.79.227.26:4653 , piece_id=JNAYIWG2T6XR5SPVQT6AKUYCORCKAL7VI62NPTOZDGA2GLMAGCFA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.586880885 (375h34m54.626454005s) [noise=true , address=145.79.227.24:4604 , piece_id=PAMMKTED4RMO432ZFCSCZI4DS5UFC6SROO3BZUKBGUXCXLJBWK7Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.590035011 (375h34m54.629608141s) [noise=true , address=145.79.227.43:4166 , piece_id=KY3MDW27JBIPAFCCVWI4DJ4NML45C3NRYQSUMRGRWQB2VQ2NFLAQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.596270615 (375h34m54.635843745s) [noise=true , address=145.79.227.45:4231 , piece_id=SNVIZK5SFCJ7M66ZW7OUHYPN7X72UKCW6WAARK2IIBJ77HMLMPRQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.599042323 (375h34m54.638615453s) [noise=true , address=145.79.227.40:4068 , piece_id=3JTAFMRWTL55NS6QX73NB63MUKOSEAXTMAELJJBMV2G3JVNE2LQA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.599626135 (375h34m54.639199255s) [noise=true , address=145.79.227.23:4578 , piece_id=BVMH3AL2WSIGAXNODTTVEMC3HSC5YL73V6PRMKOANCO7EMUNVUBA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.600114044 (375h34m54.639687164s) [noise=true , address=145.79.227.33:4898 , piece_id=5I4XYNRZ2NJI3KNJL2INORWDUTLMDR5NUGWAWAOHQRVN6R4WPDCA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.609043928 (375h34m54.648617048s) [noise=true , address=145.79.227.48:4334 , piece_id=AOGK2ALI4HDA5I624I7ASBBSHA7W6RJJ3XSSHX5IRU2GJWBXRS7A, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.610941471 (375h34m54.650514591s) [noise=true , address=145.79.227.49:4342 , piece_id=3EYT37ETP6Q74YRTJRNJ4JXGD2EVHOSUEWV4KG5XY3FTYYQSS3CQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.697140621 (375h34m54.736713751s) [noise=true , address=77.90.31.19:4862 , piece_id=XF2X3JOPDY3F56CBLS7NSCMXQDFKFFFEH7FBOFGR5XOWVDTDCWYA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.731299677 (375h34m54.770872797s) [noise=true , address=77.90.31.13:4653 , piece_id=WUJNKDL6GVG26N4ALSNIQ6TEWEPEUTUMT6GP4ZXATBCGH24MHAXQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.747909485 (375h34m54.787482615s) [noise=true , address=77.90.31.11:4566 , piece_id=UC2O2R5VQNGBGSSRMK2FOBTVGW3PQB37654YPBRKRMPD3MNM3NVA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:21.233323656 (375h34m57.272896786s) [noise=true , address=145.79.227.3:4162 , piece_id=Y4ZNYFYUV4VSWYGP3ADJUA43P3XOWT3WCGP2A7H5W5PAZWTL5Z7Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:145.79.227.3:4162): protocol: rpc: dial tcp 145.79.227.3:4162: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Sep 22 05:04:15.475732793 (375h35m51.515305923s) [noise=true , address=145.79.227.15:4878 , piece_id=SYE46YDTLQQRC2Q2CFCMHG7PDNDK5KIK6PAL7NFEFTJ5FVLGWMOQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:145.79.227.15:4878): protocol: rpc: dial tcp 145.79.227.15:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475808366 (375h35m51.515381496s) [noise=true , address=145.79.227.13:4835 , piece_id=EXPL7JZEQJBAVZ7R7H4LZ2XGQ2FEILEXKNG3CQCVIM73IE7KRUNQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:145.79.227.13:4835): protocol: rpc: dial tcp 145.79.227.13:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475871847 (375h35m51.515444977s) [noise=true , address=145.79.227.18:4034 , piece_id=HLWHBRLTVVDMAZMXCTXPPLQYHQYSWENJPXZUDQYH2SKG27EYN75Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY, address:145.79.227.18:4034): protocol: rpc: dial tcp 145.79.227.18:4034: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475936840 (375h35m51.51550997s) [noise=true , address=145.79.227.12:4791 , piece_id=FOABF7GEXKWDFSNMX72VROVITWLPI4SA3UKVEWZH64I5FLMCJDHQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:1MktgfjDE2jQUoUYAx1CVSjHjAkF8SZTv8a87Y7hbEs3RFCmgn, address:145.79.227.12:4791): protocol: rpc: dial tcp 145.79.227.12:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475978570 (375h35m51.515551701s) [seg_pos={0 0}, upload=330124]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:04:15.733369573 (375h35m51.772942703s) [upload=330124]: Done uploading segments...
Sep 22 05:04:15.733407556 (375h35m51.772980686s) [upload=330124]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.18970876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.259715249s
FAIL (2025-09-22 05:04:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
*pb.BatchRequestItem_ObjectBegin
Sep 22 05:03:16.206590601 (375h34m52.246163731s) [upload=330124]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:03:16.705371709 (375h34m52.74494485s) [seg_pos={0 0}, upload=330124]: Begin upload segment...
Sep 22 05:03:16.705448145 (375h34m52.745021275s) [seg_pos={0 0}, upload=330124]: Done begin upload segment.
Sep 22 05:03:16.705476790 (375h34m52.74504992s) [noise=true , address=145.79.227.49:4342 , piece_id=3EYT37ETP6Q74YRTJRNJ4JXGD2EVHOSUEWV4KG5XY3FTYYQSS3CQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.705488011 (375h34m52.745061141s) [noise=true , address=145.79.227.13:4835 , piece_id=EXPL7JZEQJBAVZ7R7H4LZ2XGQ2FEILEXKNG3CQCVIM73IE7KRUNQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706086350 (375h34m52.74565948s) [noise=true , address=145.79.227.26:4653 , piece_id=JNAYIWG2T6XR5SPVQT6AKUYCORCKAL7VI62NPTOZDGA2GLMAGCFA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706199506 (375h34m52.745772626s) [noise=true , address=145.79.227.12:4791 , piece_id=FOABF7GEXKWDFSNMX72VROVITWLPI4SA3UKVEWZH64I5FLMCJDHQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.706548150 (375h34m52.74612128s) [noise=true , address=145.79.227.43:4166 , piece_id=KY3MDW27JBIPAFCCVWI4DJ4NML45C3NRYQSUMRGRWQB2VQ2NFLAQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707075163 (375h34m52.746648293s) [noise=true , address=145.79.227.40:4068 , piece_id=3JTAFMRWTL55NS6QX73NB63MUKOSEAXTMAELJJBMV2G3JVNE2LQA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707425580 (375h34m52.74699871s) [noise=true , address=145.79.227.24:4604 , piece_id=PAMMKTED4RMO432ZFCSCZI4DS5UFC6SROO3BZUKBGUXCXLJBWK7Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.707776068 (375h34m52.747349188s) [noise=true , address=145.79.227.3:4162 , piece_id=Y4ZNYFYUV4VSWYGP3ADJUA43P3XOWT3WCGP2A7H5W5PAZWTL5Z7Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.708123609 (375h34m52.74769674s) [noise=true , address=145.79.227.45:4231 , piece_id=SNVIZK5SFCJ7M66ZW7OUHYPN7X72UKCW6WAARK2IIBJ77HMLMPRQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.708741957 (375h34m52.748315087s) [noise=true , address=77.90.31.13:4653 , piece_id=WUJNKDL6GVG26N4ALSNIQ6TEWEPEUTUMT6GP4ZXATBCGH24MHAXQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709385462 (375h34m52.748958602s) [noise=true , address=145.79.227.23:4578 , piece_id=BVMH3AL2WSIGAXNODTTVEMC3HSC5YL73V6PRMKOANCO7EMUNVUBA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709465565 (375h34m52.749038695s) [noise=true , address=145.79.227.15:4878 , piece_id=SYE46YDTLQQRC2Q2CFCMHG7PDNDK5KIK6PAL7NFEFTJ5FVLGWMOQ, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709863723 (375h34m52.749436853s) [noise=true , address=77.90.31.11:4566 , piece_id=UC2O2R5VQNGBGSSRMK2FOBTVGW3PQB37654YPBRKRMPD3MNM3NVA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.709882649 (375h34m52.74945578s) [noise=true , address=77.90.31.19:4862 , piece_id=XF2X3JOPDY3F56CBLS7NSCMXQDFKFFFEH7FBOFGR5XOWVDTDCWYA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.710501858 (375h34m52.750074988s) [noise=true , address=145.79.227.48:4334 , piece_id=AOGK2ALI4HDA5I624I7ASBBSHA7W6RJJ3XSSHX5IRU2GJWBXRS7A, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.710978395 (375h34m52.750551526s) [noise=true , address=145.79.227.4:4201 , piece_id=PDFMVMTXQ6TYOFLYRR4R34E7EUFI3S24NFW5SARZWTVBB2BMKO6Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.711020215 (375h34m52.750593345s) [noise=true , address=145.79.227.33:4898 , piece_id=5I4XYNRZ2NJI3KNJL2INORWDUTLMDR5NUGWAWAOHQRVN6R4WPDCA, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:16.711453480 (375h34m52.751026611s) [noise=true , address=145.79.227.18:4034 , piece_id=HLWHBRLTVVDMAZMXCTXPPLQYHQYSWENJPXZUDQYH2SKG27EYN75Q, seg_pos={0 0}, upload=330124]: Uploading piece...
Sep 22 05:03:17.877964509 (375h34m53.917537639s) [noise=true , address=145.79.227.4:4201 , piece_id=PDFMVMTXQ6TYOFLYRR4R34E7EUFI3S24NFW5SARZWTVBB2BMKO6Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:145.79.227.4:4201): protocol: rpc: dial tcp 145.79.227.4:4201: connect: no route to host; conn closed: op
Sep 22 05:03:18.187226354 (375h34m54.226799484s) [upload=330124]: Next returned nil segment
Sep 22 05:03:18.187247254 (375h34m54.226820384s) [upload=330124]: Waiting for error group managing segments...
Sep 22 05:03:18.586189148 (375h34m54.625762268s) [noise=true , address=145.79.227.26:4653 , piece_id=JNAYIWG2T6XR5SPVQT6AKUYCORCKAL7VI62NPTOZDGA2GLMAGCFA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.586880885 (375h34m54.626454005s) [noise=true , address=145.79.227.24:4604 , piece_id=PAMMKTED4RMO432ZFCSCZI4DS5UFC6SROO3BZUKBGUXCXLJBWK7Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.590035011 (375h34m54.629608141s) [noise=true , address=145.79.227.43:4166 , piece_id=KY3MDW27JBIPAFCCVWI4DJ4NML45C3NRYQSUMRGRWQB2VQ2NFLAQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.596270615 (375h34m54.635843745s) [noise=true , address=145.79.227.45:4231 , piece_id=SNVIZK5SFCJ7M66ZW7OUHYPN7X72UKCW6WAARK2IIBJ77HMLMPRQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.599042323 (375h34m54.638615453s) [noise=true , address=145.79.227.40:4068 , piece_id=3JTAFMRWTL55NS6QX73NB63MUKOSEAXTMAELJJBMV2G3JVNE2LQA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.599626135 (375h34m54.639199255s) [noise=true , address=145.79.227.23:4578 , piece_id=BVMH3AL2WSIGAXNODTTVEMC3HSC5YL73V6PRMKOANCO7EMUNVUBA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.600114044 (375h34m54.639687164s) [noise=true , address=145.79.227.33:4898 , piece_id=5I4XYNRZ2NJI3KNJL2INORWDUTLMDR5NUGWAWAOHQRVN6R4WPDCA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.609043928 (375h34m54.648617048s) [noise=true , address=145.79.227.48:4334 , piece_id=AOGK2ALI4HDA5I624I7ASBBSHA7W6RJJ3XSSHX5IRU2GJWBXRS7A, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.610941471 (375h34m54.650514591s) [noise=true , address=145.79.227.49:4342 , piece_id=3EYT37ETP6Q74YRTJRNJ4JXGD2EVHOSUEWV4KG5XY3FTYYQSS3CQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.697140621 (375h34m54.736713751s) [noise=true , address=77.90.31.19:4862 , piece_id=XF2X3JOPDY3F56CBLS7NSCMXQDFKFFFEH7FBOFGR5XOWVDTDCWYA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.731299677 (375h34m54.770872797s) [noise=true , address=77.90.31.13:4653 , piece_id=WUJNKDL6GVG26N4ALSNIQ6TEWEPEUTUMT6GP4ZXATBCGH24MHAXQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:18.747909485 (375h34m54.787482615s) [noise=true , address=77.90.31.11:4566 , piece_id=UC2O2R5VQNGBGSSRMK2FOBTVGW3PQB37654YPBRKRMPD3MNM3NVA, seg_pos={0 0}, upload=330124]: Done uploading piece. err: <nil>
Sep 22 05:03:21.233323656 (375h34m57.272896786s) [noise=true , address=145.79.227.3:4162 , piece_id=Y4ZNYFYUV4VSWYGP3ADJUA43P3XOWT3WCGP2A7H5W5PAZWTL5Z7Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:145.79.227.3:4162): protocol: rpc: dial tcp 145.79.227.3:4162: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op
Sep 22 05:04:15.475732793 (375h35m51.515305923s) [noise=true , address=145.79.227.15:4878 , piece_id=SYE46YDTLQQRC2Q2CFCMHG7PDNDK5KIK6PAL7NFEFTJ5FVLGWMOQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:145.79.227.15:4878): protocol: rpc: dial tcp 145.79.227.15:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475808366 (375h35m51.515381496s) [noise=true , address=145.79.227.13:4835 , piece_id=EXPL7JZEQJBAVZ7R7H4LZ2XGQ2FEILEXKNG3CQCVIM73IE7KRUNQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12WYQwhW2KDjDyweEx5FLYRkGwvW15f6eVHzbdrysiHeuKNtaoN, address:145.79.227.13:4835): protocol: rpc: dial tcp 145.79.227.13:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475871847 (375h35m51.515444977s) [noise=true , address=145.79.227.18:4034 , piece_id=HLWHBRLTVVDMAZMXCTXPPLQYHQYSWENJPXZUDQYH2SKG27EYN75Q, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:12jHHvRhcKfV1ufn7PRfnPHJ3ZxWJ8JUutf772G7YcKQjbpE3ZY, address:145.79.227.18:4034): protocol: rpc: dial tcp 145.79.227.18:4034: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475936840 (375h35m51.51550997s) [noise=true , address=145.79.227.12:4791 , piece_id=FOABF7GEXKWDFSNMX72VROVITWLPI4SA3UKVEWZH64I5FLMCJDHQ, seg_pos={0 0}, upload=330124]: Done uploading piece. err: ecclient: upload failed (node:1MktgfjDE2jQUoUYAx1CVSjHjAkF8SZTv8a87Y7hbEs3RFCmgn, address:145.79.227.12:4791): protocol: rpc: dial tcp 145.79.227.12:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:04:15.475978570 (375h35m51.515551701s) [seg_pos={0 0}, upload=330124]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Sep 22 05:04:15.733369573 (375h35m51.772942703s) [upload=330124]: Done uploading segments...
Sep 22 05:04:15.733407556 (375h35m51.772980686s) [upload=330124]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.259749493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [....X.X.X..X.......X.XX.X........XX.........................] duration:
6.240865304s
FAIL (2025-09-22 05:03:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.240904468s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X...X......X.X....X.....X....X.....X.X....XX............XX] duration:
30.24896041s
FAIL (2025-09-22 05:03:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Sep 22 05:03:14.844008014 (375h34m50.883581144s) [upload=330123]: Starting upload
Sep 22 05:03:14.844163720 (375h34m50.88373685s) [upload=330123]: Done starting upload
Sep 22 05:03:14.846008292 (375h34m50.885581422s) [upload=330123]: Uploading segments...
Sep 22 05:03:14.846089937 (375h34m50.885663077s) [upload=330123]: Got next segment. Inline: false
Sep 22 05:03:14.846133301 (375h34m50.885706431s) [upload=330123]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:03:14.846139973 (375h34m50.885713103s) [upload=330123]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:03:15.334586702 (375h34m51.374159833s) [seg_pos={0 0}, upload=330123]: Begin upload segment...
Sep 22 05:03:15.334652929 (375h34m51.374226059s) [seg_pos={0 0}, upload=330123]: Done begin upload segment.
Sep 22 05:03:15.334728252 (375h34m51.374301412s) [noise=true , address=145.79.227.18:4005 , piece_id=OWBTQ24OHLDDTBPEGPT4HQGTVEUP2U4XQNYVYDC6PCTMDWG3GYBQ, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.334791933 (375h34m51.374365073s) [noise=true , address=145.79.227.6:4259 , piece_id=JR2J2FVKZEBJ57HFXSRXPA2RRY7NKKLQVFJVXE6ZUTB34TOQKLWA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.335420801 (375h34m51.374993931s) [noise=true , address=45.91.116.7:4011 , piece_id=HV5YJJFDQCE7WUWLEGZDCBBDR4ACQMHQGFVQLKYWISWBJ7SXWG2Q, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.335568091 (375h34m51.375141221s) [noise=true , address=45.91.116.13:4155 , piece_id=YLUDF4Z2JJAFKGEF3PAFPT5FDXMRMUA6WQF7EEEC4TV7IAFQLVSQ, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.336420955 (375h34m51.375994086s) [noise=true , address=45.91.116.14:4190 , piece_id=MFZU75VZSGV5WFKCHCU2XIFBP4PDSZ4CL2TCKE5XG5BMB7SLTKUQ, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.336988185 (375h34m51.376561315s) [noise=true , address=45.91.116.7:4010 , piece_id=J5HXNUW4LBAWKQHDVX3LFTVRKSXSX7RWCQQFZCURAAAHZSQDXSCA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.337341728 (375h34m51.376914858s) [noise=true , address=145.79.227.17:4972 , piece_id=CWMRW3EOBMOQHD5DXVJPGY362ZLTM4C73Q65263YR2OHJRENSA5Q, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.337492396 (375h34m51.377065526s) [noise=true , address=45.91.116.14:4189 , piece_id=FOESVCO4FLCGXQJE2YFOICSNTFIRIU3M4HGIDHYZ4HYZJUUC7AEQ, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.337964975 (375h34m51.377538105s) [noise=true , address=145.79.227.11:4729 , piece_id=LCUDRBLT67EYLDYWABMOLKP6TVBNR6PN2BM53HHXH64GOBWUESBA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.338034848 (375h34m51.377607968s) [noise=true , address=45.91.116.5:4298 , piece_id=NKBZJM2KE4H75QORH4HJVJSMMYI2IIGKWQYZIJNQ6NPWAC3PNBBA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.338438327 (375h34m51.378011457s) [noise=true , address=77.90.31.10:4519 , piece_id=7BGKMKQU5XVMRZJ7SDR23PVFBVPCC7VLB53JYEH7E7QXMS4H5SQQ, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.340652213 (375h34m51.380225343s) [noise=true , address=145.79.227.7:4293 , piece_id=S3S53BCXK4Z7LLSOXWXPMRL2K6ESKIGO2R7NSDHADWTPGYEQ37PA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.341028259 (375h34m51.380601379s) [noise=true , address=77.90.31.16:4733 , piece_id=RNSY3KLXJZ5OCG6QWU36D5R3B4EXYXLZ2BNPQ55FNX34B6RTYK3Q, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.341039490 (375h34m51.38061263s) [noise=true , address=145.79.227.12:4765 , piece_id=ENDVBQVZHCS3ECU3IIA3LF45WMSGLC7GFSUKFFBJCSNN3SRVVPIA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.342795584 (375h34m51.382368704s) [noise=true , address=45.91.116.8:4045 , piece_id=JSO63CRRYHX2M3JD4FDU2PZOFKPC5X3C3LDDEOKS6S2E43Y3PG5A, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:15.342877150 (375h34m51.38245028s) [noise=true , address=45.91.116.9:4261 , piece_id=MLDVWAXUDJDGFUU5MHYITBZIX4MVLNR2SFZVVWUSKNVSVJYTZ5WA, seg_pos={0 0}, upload=330123]: Uploading piece...
Sep 22 05:03:17.148147993 (375h34m53.187721123s) [upload=330123]: Next returned nil segment
Sep 22 05:03:17.148172670 (375h34m53.1877458s) [upload=330123]: Waiting for error group managing segments...
Sep 22 05:03:17.435494238 (375h34m53.475067358s) [noise=true , address=45.91.116.8:4045 , piece_id=JSO63CRRYHX2M3JD4FDU2PZOFKPC5X3C3LDDEOKS6S2E43Y3PG5A, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.449454661 (375h34m53.489027791s) [noise=true , address=45.91.116.14:4190 , piece_id=MFZU75VZSGV5WFKCHCU2XIFBP4PDSZ4CL2TCKE5XG5BMB7SLTKUQ, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.450028984 (375h34m53.489602114s) [noise=true , address=77.90.31.16:4733 , piece_id=RNSY3KLXJZ5OCG6QWU36D5R3B4EXYXLZ2BNPQ55FNX34B6RTYK3Q, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.461188253 (375h34m53.500761384s) [noise=true , address=45.91.116.13:4155 , piece_id=YLUDF4Z2JJAFKGEF3PAFPT5FDXMRMUA6WQF7EEEC4TV7IAFQLVSQ, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.469052589 (375h34m53.508625719s) [noise=true , address=45.91.116.7:4011 , piece_id=HV5YJJFDQCE7WUWLEGZDCBBDR4ACQMHQGFVQLKYWISWBJ7SXWG2Q, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.479059975 (375h34m53.518633105s) [noise=true , address=45.91.116.14:4189 , piece_id=FOESVCO4FLCGXQJE2YFOICSNTFIRIU3M4HGIDHYZ4HYZJUUC7AEQ, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.483756869 (375h34m53.523329999s) [noise=true , address=45.91.116.5:4298 , piece_id=NKBZJM2KE4H75QORH4HJVJSMMYI2IIGKWQYZIJNQ6NPWAC3PNBBA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.483811473 (375h34m53.523384603s) [noise=true , address=77.90.31.10:4519 , piece_id=7BGKMKQU5XVMRZJ7SDR23PVFBVPCC7VLB53JYEH7E7QXMS4H5SQQ, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.488535669 (375h34m53.528108809s) [noise=true , address=45.91.116.9:4261 , piece_id=MLDVWAXUDJDGFUU5MHYITBZIX4MVLNR2SFZVVWUSKNVSVJYTZ5WA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:17.533266786 (375h34m53.572839916s) [noise=true , address=45.91.116.7:4010 , piece_id=J5HXNUW4LBAWKQHDVX3LFTVRKSXSX7RWCQQFZCURAAAHZSQDXSCA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: <nil>
Sep 22 05:03:44.137307074 (375h35m20.176880194s) [noise=true , address=145.79.227.17:4972 , piece_id=CWMRW3EOBMOQHD5DXVJPGY362ZLTM4C73Q65263YR2OHJRENSA5Q, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137363953 (375h35m20.176937073s) [noise=true , address=145.79.227.6:4259 , piece_id=JR2J2FVKZEBJ57HFXSRXPA2RRY7NKKLQVFJVXE6ZUTB34TOQKLWA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137430869 (375h35m20.177003999s) [noise=true , address=145.79.227.18:4005 , piece_id=OWBTQ24OHLDDTBPEGPT4HQGTVEUP2U4XQNYVYDC6PCTMDWG3GYBQ, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137495512 (375h35m20.177068652s) [noise=true , address=145.79.227.12:4765 , piece_id=ENDVBQVZHCS3ECU3IIA3LF45WMSGLC7GFSUKFFBJCSNN3SRVVPIA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: rpc: dial tcp 145.79.227.12:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137531661 (375h35m20.177104801s) [noise=true , address=145.79.227.11:4729 , piece_id=LCUDRBLT67EYLDYWABMOLKP6TVBNR6PN2BM53HHXH64GOBWUESBA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137595343 (375h35m20.177168473s) [noise=true , address=145.79.227.7:4293 , piece_id=S3S53BCXK4Z7LLSOXWXPMRL2K6ESKIGO2R7NSDHADWTPGYEQ37PA, seg_pos={0 0}, upload=330123]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:44.137654686 (375h35m20.177227816s) [seg_pos={0 0}, upload=330123]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Sep 22 05:03:44.384132058 (375h35m20.423705188s) [upload=330123]: Done uploading segments...
Sep 22 05:03:44.384168828 (375h35m20.423741958s) [upload=330123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.248978303s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.....X.X.X..X.......X.XX.X........XX........................] duration:
5.997929133s
FAIL (2025-09-22 05:03:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.998044923s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X...X......X.X....X.....X....X.....X.X....XX............X] duration:
30.244548656s
FAIL (2025-09-22 05:03:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 05:02:40.373445830 (375h34m16.41301896s) [upload=330118]: Starting upload
Sep 22 05:02:40.373524019 (375h34m16.413097139s) [upload=330118]: Done starting upload
Sep 22 05:02:40.373579615 (375h34m16.413152735s) [upload=330118]: Uploading segments...
Sep 22 05:02:40.373735070 (375h34m16.41330819s) [upload=330118]: Got next segment. Inline: false
Sep 22 05:02:40.373764597 (375h34m16.413337717s) [upload=330118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 05:02:40.373770778 (375h34m16.413343908s) [upload=330118]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 05:02:40.623600554 (375h34m16.663173674s) [seg_pos={0 0}, upload=330118]: Begin upload segment...
Sep 22 05:02:40.623645789 (375h34m16.663218919s) [seg_pos={0 0}, upload=330118]: Done begin upload segment.
Sep 22 05:02:40.623670767 (375h34m16.663243887s) [noise=true , address=77.90.31.14:4663 , piece_id=ENMUBV6GTZICHF5K4UGFHJARHWHICIHDYAF2E4S6NIEH7ABGIBVQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.623686388 (375h34m16.663259518s) [noise=true , address=145.79.227.11:4732 , piece_id=G47RATJAKKSHIH5PXWKNUE3Q4HBXKL54J24QSGISQ3HFUARRXP2A, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624009102 (375h34m16.663582222s) [noise=true , address=145.79.227.3:4149 , piece_id=XNMSYRCCZMVWGZQSJ3WSG3SBRHN2IP4DVSVKZG6W45SF2SKC2QSQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624017247 (375h34m16.663590367s) [noise=true , address=77.90.31.12:4589 , piece_id=EXJE3NQ5XGPGLVQJJ2V3AUEWOZBF5NVTHONOZTMGGA42KWPDQ6KA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624267463 (375h34m16.663840593s) [noise=true , address=77.90.31.16:4734 , piece_id=56PL2NF5J6LTRR2V2BMSHKYB7GAE7VR3FZ47IKFYJHFKVJJJOLTA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624424252 (375h34m16.663997382s) [noise=true , address=45.91.116.12:4226 , piece_id=HWZHYJTUAO55WTRFVUS7C5LTFFWVCZLY52RAOOLZAUXXXMR5A3AQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624667656 (375h34m16.664240786s) [noise=true , address=45.91.116.8:4045 , piece_id=PAQMWD4SPJXYUMHOGYKCJCBCNXQFY7PDZDVGFWX5NP2WVQJR5K5A, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.624775090 (375h34m16.66434821s) [noise=true , address=77.90.31.13:4627 , piece_id=I7UTXOM56RRUHFLBBOWNUBVSZXXRWB23IPGZ5KNQI2L3BBEJGIBQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625017342 (375h34m16.664590462s) [noise=true , address=145.79.227.18:4006 , piece_id=OLAUPTF5YDOGT6NLYDKAWFI4UZG7SLFT47SSLPU3RIYMKV56M4MA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625073970 (375h34m16.6646471s) [noise=true , address=45.91.116.13:4153 , piece_id=77AR4DM4GIHT37DSVVG3P4QYNM527GVWS4HNVWS7M7EDSGNYF5UA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625291193 (375h34m16.664864323s) [noise=true , address=45.91.116.4:4938 , piece_id=WU4ZFJMLSKW7JX2GHK6522KQGTOOVAZNAXM4JYUE4PD3XBLGWQPA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625314428 (375h34m16.664887558s) [noise=true , address=45.91.116.14:4190 , piece_id=ORWLWLNYMYTYS2BFRT5XHEBSZLDQW32BMA3DF55YKV5EVWNDGKHQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625559054 (375h34m16.665132184s) [noise=true , address=77.90.31.10:4519 , piece_id=72FLOBB6CO5TELT3KELLM37TDYDQJ3L57ZBG7VS65EZX23YHHPZQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.625880686 (375h34m16.665453806s) [noise=true , address=145.79.227.4:4187 , piece_id=GZOPC5CLN2CBNSGPZP2CMJT5G52FGVFIR5KAXA2HBQJ7TD2TDCRA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.626163594 (375h34m16.665736724s) [noise=true , address=45.91.116.12:4227 , piece_id=AM5JJG2LZCSECRVIWFYNBRZGIUJI354OZVCDSSRLMEKJYXEODRKQ, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:40.627003704 (375h34m16.666576824s) [noise=true , address=45.91.116.5:4297 , piece_id=EJ6AFD2FAHDOFSKTLU47B33UHHUZDHW4YLLGHWKW4YKVCW7CYOKA, seg_pos={0 0}, upload=330118]: Uploading piece...
Sep 22 05:02:42.393328819 (375h34m18.432901939s) [upload=330118]: Next returned nil segment
Sep 22 05:02:42.393347726 (375h34m18.432920846s) [upload=330118]: Waiting for error group managing segments...
Sep 22 05:02:42.693452323 (375h34m18.733025443s) [noise=true , address=45.91.116.8:4045 , piece_id=PAQMWD4SPJXYUMHOGYKCJCBCNXQFY7PDZDVGFWX5NP2WVQJR5K5A, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.725133220 (375h34m18.76470636s) [noise=true , address=45.91.116.14:4190 , piece_id=ORWLWLNYMYTYS2BFRT5XHEBSZLDQW32BMA3DF55YKV5EVWNDGKHQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.734683085 (375h34m18.774256216s) [noise=true , address=45.91.116.4:4938 , piece_id=WU4ZFJMLSKW7JX2GHK6522KQGTOOVAZNAXM4JYUE4PD3XBLGWQPA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.743328128 (375h34m18.782901258s) [noise=true , address=77.90.31.14:4663 , piece_id=ENMUBV6GTZICHF5K4UGFHJARHWHICIHDYAF2E4S6NIEH7ABGIBVQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.751160352 (375h34m18.790733482s) [noise=true , address=45.91.116.12:4227 , piece_id=AM5JJG2LZCSECRVIWFYNBRZGIUJI354OZVCDSSRLMEKJYXEODRKQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.781120872 (375h34m18.820694002s) [noise=true , address=45.91.116.5:4297 , piece_id=EJ6AFD2FAHDOFSKTLU47B33UHHUZDHW4YLLGHWKW4YKVCW7CYOKA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.786774949 (375h34m18.826348079s) [noise=true , address=77.90.31.13:4627 , piece_id=I7UTXOM56RRUHFLBBOWNUBVSZXXRWB23IPGZ5KNQI2L3BBEJGIBQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.799365233 (375h34m18.838938353s) [noise=true , address=45.91.116.12:4226 , piece_id=HWZHYJTUAO55WTRFVUS7C5LTFFWVCZLY52RAOOLZAUXXXMR5A3AQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.814983724 (375h34m18.854556854s) [noise=true , address=77.90.31.10:4519 , piece_id=72FLOBB6CO5TELT3KELLM37TDYDQJ3L57ZBG7VS65EZX23YHHPZQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.853033488 (375h34m18.892606608s) [noise=true , address=77.90.31.12:4589 , piece_id=EXJE3NQ5XGPGLVQJJ2V3AUEWOZBF5NVTHONOZTMGGA42KWPDQ6KA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.876511004 (375h34m18.916084124s) [noise=true , address=77.90.31.16:4734 , piece_id=56PL2NF5J6LTRR2V2BMSHKYB7GAE7VR3FZ47IKFYJHFKVJJJOLTA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:02:42.917577232 (375h34m18.957150362s) [noise=true , address=45.91.116.13:4153 , piece_id=77AR4DM4GIHT37DSVVG3P4QYNM527GVWS4HNVWS7M7EDSGNYF5UA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: <nil>
Sep 22 05:03:09.663238313 (375h34m45.702811443s) [noise=true , address=145.79.227.18:4006 , piece_id=OLAUPTF5YDOGT6NLYDKAWFI4UZG7SLFT47SSLPU3RIYMKV56M4MA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:09.663337622 (375h34m45.702910752s) [noise=true , address=145.79.227.11:4732 , piece_id=G47RATJAKKSHIH5PXWKNUE3Q4HBXKL54J24QSGISQ3HFUARRXP2A, seg_pos={0 0}, upload=330118]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:09.663393429 (375h34m45.702966559s) [noise=true , address=145.79.227.4:4187 , piece_id=GZOPC5CLN2CBNSGPZP2CMJT5G52FGVFIR5KAXA2HBQJ7TD2TDCRA, seg_pos={0 0}, upload=330118]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 05:03:09.663480374 (375h34m45.703053504s) [noise=true , address=145.79.227.3:4149 , piece_id=XNMSYRCCZMVWGZQSJ3WSG3SBRHN2IP4DVSVKZG6W45SF2SKC2QSQ, seg_pos={0 0}, upload=330118]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Sep 22 05:03:09.663535048 (375h34m45.70310877s) [seg_pos={0 0}, upload=330118]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 05:03:09.906318684 (375h34m45.945891814s) [upload=330118]: Done uploading segments...
Sep 22 05:03:09.906357167 (375h34m45.945930297s) [upload=330118]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.244576399s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [......X.X.X..X.......X.XX.X........XX.......................] duration:
6.019653235s
FAIL (2025-09-22 05:03:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.019681578s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X......X.X.X..X.......X.XX.X........XX......................] duration:
6.068001294s
FAIL (2025-09-22 05:02:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.068036792s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....X...X......X.X....X.....X....X.....X.X....XX............] duration:
6.121323179s
FAIL (2025-09-22 05:02:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.121360108s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X......X.X.X..X.......X.XX.X........XX.....................] duration:
6.200626712s
FAIL (2025-09-22 05:01:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.200654956s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....X...X......X.X....X.....X....X.....X.X....XX...........] duration:
6.083647062s
FAIL (2025-09-22 05:00:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.083677308s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X.X......X.X.X..X.......X.XX.X........XX....................] duration:
6.495845427s
FAIL (2025-09-22 05:00:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.495876807s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X.....X...X......X.X....X.....X....X.....X.X....XX..........] duration:
6.848794552s
FAIL (2025-09-22 05:00:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.848847163s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [XX.X......X.X.X..X.......X.XX.X........XX...................] duration:
6.323937426s
FAIL (2025-09-22 05:00:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.323964158s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X.....X...X......X.X....X.....X....X.....X.X....XX.........] duration:
6.121662528s
FAIL (2025-09-22 04:59:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.121694848s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..X.....X...X......X.X....X.....X....X.....X.X....XX........] duration:
6.211620234s
FAIL (2025-09-22 04:59:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.211654319s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [XXX.X......X.X.X..X.......X.XX.X........XX..................] duration:
5.994173163s
FAIL (2025-09-22 04:58:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.99420381s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X.....X...X......X.X....X.....X....X.....X.X....XX.......] duration:
6.048952914s
FAIL (2025-09-22 04:58:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.0490034s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [....X.....X...X......X.X....X.....X....X.....X.X....XX......] duration:
6.133774222s
FAIL (2025-09-22 04:58:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.13380475s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XXXX.X......X.X.X..X.......X.XX.X........XX.................] duration:
5.98497529s
FAIL (2025-09-22 04:58:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.985003265s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XXXXX.X......X.X.X..X.......X.XX.X........XX................] duration:
6.032448336s
FAIL (2025-09-22 04:57:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.032476619s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.....X.....X...X......X.X....X.....X....X.....X.X....XX.....] duration:
6.192362263s
FAIL (2025-09-22 04:57:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.192394864s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXX.X......X.X.X..X.......X.XX.X........XX...............] duration:
6.169755305s
FAIL (2025-09-22 04:57:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.169780592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [......X.....X...X......X.X....X.....X....X.....X.X....XX....] duration:
6.330775016s
FAIL (2025-09-22 04:56:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.330941323s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXX.X......X.X.X..X.......X.XX.X........XX..............] duration:
6.0293885s
FAIL (2025-09-22 04:56:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.0294152s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.......X.....X...X......X.X....X.....X....X.....X.X....XX...] duration:
6.18042306s
FAIL (2025-09-22 04:55:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.180458948s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XXXXXXX.X......X.X.X..X.......X.XX.X........XX.............] duration:
6.107852369s
FAIL (2025-09-22 04:55:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.107881535s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..XXXXXXX.X......X.X.X..X.......X.XX.X........XX............] duration:
5.995272746s
FAIL (2025-09-22 04:54:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.995296892s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [........X.....X...X......X.X....X.....X....X.....X.X....XX..] duration:
5.986088085s
FAIL (2025-09-22 04:54:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.986121146s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...XXXXXXX.X......X.X.X..X.......X.XX.X........XX...........] duration:
6.036392415s
FAIL (2025-09-22 04:54:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.036424266s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X........X.....X...X......X.X....X.....X....X.....X.X....XX.] duration:
6.21549405s
FAIL (2025-09-22 04:53:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.215549835s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [XX........X.....X...X......X.X....X.....X....X.....X.X....XX] duration:
30.239229431s
FAIL (2025-09-22 04:53:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:53:02.850419531 (375h24m38.889992661s) [upload=329978]: Starting upload
Sep 22 04:53:02.850522287 (375h24m38.890095407s) [upload=329978]: Done starting upload
Sep 22 04:53:02.850680618 (375h24m38.890253748s) [upload=329978]: Uploading segments...
Sep 22 04:53:02.850789495 (375h24m38.890362615s) [upload=329978]: Got next segment. Inline: false
Sep 22 04:53:02.850885137 (375h24m38.890458267s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:53:02.850894956 (375h24m38.890468086s) [upload=329978]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:53:03.094738790 (375h24m39.13431192s) [seg_pos={0 0}, upload=329978]: Begin upload segment...
Sep 22 04:53:03.094819152 (375h24m39.134392282s) [seg_pos={0 0}, upload=329978]: Done begin upload segment.
Sep 22 04:53:03.094893134 (375h24m39.134466264s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095441638 (375h24m39.135014778s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095468279 (375h24m39.135041399s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.095808817 (375h24m39.135381948s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096106965 (375h24m39.135680095s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096614852 (375h24m39.136187982s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.096651291 (375h24m39.136224422s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097014122 (375h24m39.136587262s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097103352 (375h24m39.136676482s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.097580992 (375h24m39.137154122s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.098127913 (375h24m39.137701043s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102189228 (375h24m39.141762358s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.102963592 (375h24m39.142536722s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103166859 (375h24m39.142739989s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.103894475 (375h24m39.143467605s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:03.105267469 (375h24m39.144840599s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Uploading piece...
Sep 22 04:53:04.864711776 (375h24m40.904284906s) [upload=329978]: Next returned nil segment
Sep 22 04:53:04.864811276 (375h24m40.904384406s) [upload=329978]: Waiting for error group managing segments...
Sep 22 04:53:05.146785038 (375h24m41.186358158s) [noise=true , address=77.90.31.17:4772 , piece_id=O4UNU5LWEJGQABHDKQD77IO3U6I5SDE2I3UZICIUL4KOORIW6Q6A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.168864120 (375h24m41.20843725s) [noise=true , address=45.91.116.10:4081 , piece_id=LNBEXBWLPZ2UFRAVM6NQUKGXA2D23R3PYBXD4UEACNYDMIBEUWGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.186516585 (375h24m41.226089715s) [noise=true , address=45.91.116.12:4225 , piece_id=ZGVZNWOLQW2UMPSZORM7M3UNOPVQVN3I6SYOS732DTIKG74JB3YA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.211426808 (375h24m41.250999938s) [noise=true , address=77.90.31.19:4841 , piece_id=HQVEK5SR5SUEQM4G6EDGUJHZIXOIVBSAS3IRRVUKU5W22KV6PPCA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.225930857 (375h24m41.265503987s) [noise=true , address=45.91.116.11:4117 , piece_id=TR4LOT5NRLAZDKMZHU3LJLTUBCK77FA4WVDZSGLUXXNUMVXADFUA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.241197037 (375h24m41.280770167s) [noise=true , address=77.90.31.16:4734 , piece_id=GCE5E3LVQR6WF6ML5P3TO2IGIP5R6N3HIJD5UJW6TITI42FI3JKA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.244220685 (375h24m41.283793815s) [noise=true , address=77.90.31.18:4806 , piece_id=V6Q7SOIP5YOEFGZFPIEMUXLELENND2ZTHTULHDP2UOYDHGHS6UTQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.245641008 (375h24m41.285214128s) [noise=true , address=45.91.116.9:4263 , piece_id=GX34772I4HPOI5XLVUXVFV3WM65L3E3JHUXOJA4XEY3Y5ZOEK3FQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.252009445 (375h24m41.291582575s) [noise=true , address=45.91.116.8:4048 , piece_id=G7C7Q6OZS672KA4GOC2OA7P757KPIH3NQDSN5TG2MTGLWOBCVM3A, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.290222671 (375h24m41.329795811s) [noise=true , address=45.91.116.12:4226 , piece_id=S76KEQJNUFBCMJ4SAYIOTZVE6LEXBVQYG4PMGSUH2VS57ZZ5EFXA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.300595382 (375h24m41.340168512s) [noise=true , address=77.90.31.14:4663 , piece_id=K6IGI6IASYSIXRPXZZMOFB5GVG5U4IGH4JUHOSMJE64OLGOWCVSQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:05.346353945 (375h24m41.385927075s) [noise=true , address=45.91.116.6:4975 , piece_id=OUVNSS7SGYNLEIIJX3AZCORACMQUFWCCWXSYXQT3YXEDXBB4L3RA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: <nil>
Sep 22 04:53:32.165674158 (375h25m8.205247288s) [noise=true , address=145.79.227.18:4007 , piece_id=6IOYYPMJZHCJJXJQOGYCSPS5BTWCYLOXP6FEYCXR7OZUG6IGSXGA, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165740645 (375h25m8.205313775s) [noise=true , address=145.79.227.2:4115 , piece_id=NF5ZQ7IJSYTO2X7YAUAJCNAJ4Y4TTSFZKDLAVX6PSHKECQBTGFZQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165772295 (375h25m8.205345425s) [noise=true , address=145.79.227.16:4912 , piece_id=VD2F5BXERTIFABQEKTOH4X6W3POQ4DSF7R4MAKFGZ6QMP7TN5IYQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165841447 (375h25m8.205414577s) [noise=true , address=145.79.227.4:4187 , piece_id=S7W42K7METJSBCQFB36QYL3N32D64IDUVVMXSNTWENJUE4MHK6SQ, seg_pos={0 0}, upload=329978]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:145.79.227.4:4187): protocol: rpc: dial tcp 145.79.227.4:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:53:32.165881082 (375h25m8.205454212s) [seg_pos={0 0}, upload=329978]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:53:32.402457289 (375h25m8.442030419s) [upload=329978]: Done uploading segments...
Sep 22 04:53:32.402496574 (375h25m8.442069714s) [upload=329978]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.239268246s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [....XXXXXXX.X......X.X.X..X.......X.XX.X........XX..........] duration:
6.284125012s
FAIL (2025-09-22 04:53:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.284150912s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.XX........X.....X...X......X.X....X.....X....X.....X.X....X] duration:
30.2445539s
FAIL (2025-09-22 04:52:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:52:20.406787243 (375h23m56.446360373s) [upload=329965]: Starting upload
Sep 22 04:52:20.406855042 (375h23m56.446428172s) [upload=329965]: Done starting upload
Sep 22 04:52:20.407079480 (375h23m56.44665262s) [upload=329965]: Uploading segments...
Sep 22 04:52:20.407133071 (375h23m56.446706201s) [upload=329965]: Got next segment. Inline: false
Sep 22 04:52:20.407161896 (375h23m56.446735026s) [upload=329965]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:52:20.407167186 (375h23m56.446740306s) [upload=329965]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:52:20.652758038 (375h23m56.692331158s) [seg_pos={0 0}, upload=329965]: Begin upload segment...
Sep 22 04:52:20.652855473 (375h23m56.692428603s) [seg_pos={0 0}, upload=329965]: Done begin upload segment.
Sep 22 04:52:20.652908385 (375h23m56.692481515s) [noise=true , address=45.91.116.12:4227 , piece_id=QVEQXZE2LQW4ZV2YN3OLUN4O7TME42R3ZGWNXUI4UALRKOKHDMSQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.652917251 (375h23m56.692490371s) [noise=true , address=145.79.227.3:4151 , piece_id=VRRKCY44BV2ENO7DWUV2IIT6GVDDTXLLQSHCYSTIDJEEBAMPDCDQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.653004017 (375h23m56.692577207s) [noise=true , address=145.79.227.3:4149 , piece_id=CM7CGZ3P3EI7IONX2KP7VRYJPY4D7FF2DJWNJMTZ5GBQRSCGI6TQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.653479321 (375h23m56.693052441s) [noise=true , address=77.90.31.17:4770 , piece_id=Q4W423J6DNH6UKHQQL2T7D3BIL4MABWUT3KSWXK4EHBCNWJ6UQAA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.653554324 (375h23m56.693127444s) [noise=true , address=45.91.116.6:4974 , piece_id=GJNQQKP3GUCRAACWLBFJTCEXQNHFT76HXIEZDB4E6X22GAWEX3CA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.653851861 (375h23m56.693424981s) [noise=true , address=77.90.31.19:4842 , piece_id=XGD5D2XSLR526BP2ZT536DE3HF4LBBK6Z6WVZ7HSSPZF4UXPQNCA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.653889994 (375h23m56.693463114s) [noise=true , address=77.90.31.11:4553 , piece_id=EMA6NBPWMMXY7HCCRBIPZZMLLLHN6I4E556HMC77UPMMWGP7T7HQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.654301557 (375h23m56.693874688s) [noise=true , address=77.90.31.11:4555 , piece_id=RQFOYT2JOJDXPJSZBCYTRGH6TBGZI27355OHRYC2TNV72QW4UR4A, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.654726436 (375h23m56.694299566s) [noise=true , address=77.90.31.21:4879 , piece_id=ZEH6ZAQZIEULTN4LJRLKOBNFRAAVUKGLBJLCXTAJHHAXXUNAYPPA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.656535201 (375h23m56.696108321s) [noise=true , address=45.91.116.5:4298 , piece_id=MFVZCCEJF5TAD3HR3OJ2RPQ4OGCX7RZDPMHJAABFCHP32S2QK4WA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.656865290 (375h23m56.69643841s) [noise=true , address=45.91.116.7:4011 , piece_id=S5EQ2FDXCEMM5L2KKGXOUZZL2IDQCKEVZQNSWAFUUCYB7HM3BLOA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.657203885 (375h23m56.696777015s) [noise=true , address=45.91.116.10:4083 , piece_id=AJHUGPNO3M47Q27NDK66EZXBBSLT7G2XOAJE7PVNVOHUAS2LCMXQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.658031229 (375h23m56.697604359s) [noise=true , address=145.79.227.14:4837 , piece_id=XCCLMFQVCFILUWEWQU3WNC5Z6CQ5QXBDQELYSHLA55B2NH4U2CXQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.658214419 (375h23m56.697787559s) [noise=true , address=45.91.116.11:4119 , piece_id=4DOVKQ7BSFCOANG2FMCQSP7G4EVYAHSHTAN7AUH3XTTCG2SSJ7UA, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.658496476 (375h23m56.698069606s) [noise=true , address=45.91.116.4:4939 , piece_id=OEMAEEXNUGUJPO6SZKLXJNSUJJX3AQSSHHXCUJ3RGSXHREYJA35Q, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:20.658870288 (375h23m56.698443418s) [noise=true , address=145.79.227.14:4839 , piece_id=U6HX4JCH67T3Z5BLX2KMLWZTQMUEYUUPUQN3OQTBWQMT756XAMKQ, seg_pos={0 0}, upload=329965]: Uploading piece...
Sep 22 04:52:22.340023485 (375h23m58.379596615s) [upload=329965]: Next returned nil segment
Sep 22 04:52:22.340045366 (375h23m58.379618496s) [upload=329965]: Waiting for error group managing segments...
Sep 22 04:52:22.735186551 (375h23m58.774759681s) [noise=true , address=45.91.116.7:4011 , piece_id=S5EQ2FDXCEMM5L2KKGXOUZZL2IDQCKEVZQNSWAFUUCYB7HM3BLOA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.743490453 (375h23m58.783063583s) [noise=true , address=45.91.116.12:4227 , piece_id=QVEQXZE2LQW4ZV2YN3OLUN4O7TME42R3ZGWNXUI4UALRKOKHDMSQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.743954837 (375h23m58.783527967s) [noise=true , address=45.91.116.11:4119 , piece_id=4DOVKQ7BSFCOANG2FMCQSP7G4EVYAHSHTAN7AUH3XTTCG2SSJ7UA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.744905918 (375h23m58.784479038s) [noise=true , address=45.91.116.5:4298 , piece_id=MFVZCCEJF5TAD3HR3OJ2RPQ4OGCX7RZDPMHJAABFCHP32S2QK4WA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.748872461 (375h23m58.788445581s) [noise=true , address=77.90.31.17:4770 , piece_id=Q4W423J6DNH6UKHQQL2T7D3BIL4MABWUT3KSWXK4EHBCNWJ6UQAA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.751959249 (375h23m58.791532379s) [noise=true , address=77.90.31.11:4553 , piece_id=EMA6NBPWMMXY7HCCRBIPZZMLLLHN6I4E556HMC77UPMMWGP7T7HQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.754942440 (375h23m58.79451556s) [noise=true , address=77.90.31.21:4879 , piece_id=ZEH6ZAQZIEULTN4LJRLKOBNFRAAVUKGLBJLCXTAJHHAXXUNAYPPA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.759069197 (375h23m58.798642317s) [noise=true , address=45.91.116.6:4974 , piece_id=GJNQQKP3GUCRAACWLBFJTCEXQNHFT76HXIEZDB4E6X22GAWEX3CA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.769768591 (375h23m58.809341721s) [noise=true , address=45.91.116.10:4083 , piece_id=AJHUGPNO3M47Q27NDK66EZXBBSLT7G2XOAJE7PVNVOHUAS2LCMXQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.771525366 (375h23m58.811098496s) [noise=true , address=45.91.116.4:4939 , piece_id=OEMAEEXNUGUJPO6SZKLXJNSUJJX3AQSSHHXCUJ3RGSXHREYJA35Q, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.796176567 (375h23m58.835749737s) [noise=true , address=77.90.31.19:4842 , piece_id=XGD5D2XSLR526BP2ZT536DE3HF4LBBK6Z6WVZ7HSSPZF4UXPQNCA, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:22.830118429 (375h23m58.869691549s) [noise=true , address=77.90.31.11:4555 , piece_id=RQFOYT2JOJDXPJSZBCYTRGH6TBGZI27355OHRYC2TNV72QW4UR4A, seg_pos={0 0}, upload=329965]: Done uploading piece. err: <nil>
Sep 22 04:52:49.724496993 (375h24m25.764070123s) [noise=true , address=145.79.227.14:4837 , piece_id=XCCLMFQVCFILUWEWQU3WNC5Z6CQ5QXBDQELYSHLA55B2NH4U2CXQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:52:49.724563520 (375h24m25.76413665s) [noise=true , address=145.79.227.3:4149 , piece_id=CM7CGZ3P3EI7IONX2KP7VRYJPY4D7FF2DJWNJMTZ5GBQRSCGI6TQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 22 04:52:49.724661397 (375h24m25.764234527s) [noise=true , address=145.79.227.14:4839 , piece_id=U6HX4JCH67T3Z5BLX2KMLWZTQMUEYUUPUQN3OQTBWQMT756XAMKQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; context deadline exceeded
Sep 22 04:52:49.724720510 (375h24m25.76429363s) [noise=true , address=145.79.227.3:4151 , piece_id=VRRKCY44BV2ENO7DWUV2IIT6GVDDTXLLQSHCYSTIDJEEBAMPDCDQ, seg_pos={0 0}, upload=329965]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:52:49.724746640 (375h24m25.76431976s) [seg_pos={0 0}, upload=329965]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:52:49.966619503 (375h24m26.006192633s) [upload=329965]: Done uploading segments...
Sep 22 04:52:49.966650121 (375h24m26.006223252s) [upload=329965]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.244644021s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....XXXXXXX.X......X.X.X..X.......X.XX.X........XX.........] duration:
5.986474814s
FAIL (2025-09-22 04:52:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.986509289s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [......XXXXXXX.X......X.X.X..X.......X.XX.X........XX........] duration:
6.363316432s
FAIL (2025-09-22 04:52:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.363344447s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X......XXXXXXX.X......X.X.X..X.......X.XX.X........XX.......] duration:
6.29271331s
FAIL (2025-09-22 04:51:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.292741203s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..XX........X.....X...X......X.X....X.....X....X.....X.X....] duration:
6.071694379s
FAIL (2025-09-22 04:51:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:48:51.827659682 (375h20m27.867232812s) [upload=329915]: Starting upload
Sep 22 04:48:51.827743903 (375h20m27.867317033s) [upload=329915]: Done starting upload
Sep 22 04:48:51.827860204 (375h20m27.867433334s) [upload=329915]: Uploading segments...
Sep 22 04:48:51.828124368 (375h20m27.867697488s) [upload=329915]: Got next segment. Inline: false
Sep 22 04:48:51.828212004 (375h20m27.867785124s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:48:51.828220991 (375h20m27.867794121s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:48:52.083032281 (375h20m28.122605401s) [seg_pos={0 0}, upload=329915]: Begin upload segment...
Sep 22 04:48:52.083079671 (375h20m28.122652791s) [seg_pos={0 0}, upload=329915]: Done begin upload segment.
Sep 22 04:48:52.083103035 (375h20m28.122676165s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083129847 (375h20m28.122702967s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083655047 (375h20m28.123228167s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084399634 (375h20m28.123972754s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084672223 (375h20m28.124245353s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084759530 (375h20m28.12433265s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084998324 (375h20m28.124571454s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085115347 (375h20m28.124688467s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085313976 (375h20m28.124887096s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085532622 (375h20m28.125105742s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085983340 (375h20m28.12555646s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086482561 (375h20m28.126055681s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086954470 (375h20m28.12652761s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087292853 (375h20m28.126865983s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087644342 (375h20m28.127217472s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.088021100 (375h20m28.12759422s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:53.832184080 (375h20m29.8717572s) [upload=329915]: Next returned nil segment
Sep 22 04:48:53.832208627 (375h20m29.871781757s) [upload=329915]: Waiting for error group managing segments...
Sep 22 04:48:54.111138623 (375h20m30.150711753s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.174818501 (375h20m30.214391622s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.183976781 (375h20m30.223549901s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.195801608 (375h20m30.235374728s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.205120883 (375h20m30.244694013s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.208206349 (375h20m30.247779479s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.223399449 (375h20m30.262972579s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.243643157 (375h20m30.283216277s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.263265141 (375h20m30.302838271s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.287740006 (375h20m30.327313136s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.319415021 (375h20m30.358988151s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.325619996 (375h20m30.365193117s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:49:21.116551863 (375h20m57.156124983s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116629871 (375h20m57.156203001s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116644179 (375h20m57.156217309s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116719182 (375h20m57.156292312s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116757505 (375h20m57.156330625s) [seg_pos={0 0}, upload=329915]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:49:21.360898295 (375h20m57.400471436s) [upload=329915]: Done uploading segments...
Sep 22 04:49:21.360932911 (375h20m57.400506041s) [upload=329915]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.071731269s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...XX........X.....X...X......X.X....X.....X....X.....X.X...] duration:
6.108911626s
FAIL (2025-09-22 04:51:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:48:51.827659682 (375h20m27.867232812s) [upload=329915]: Starting upload
Sep 22 04:48:51.827743903 (375h20m27.867317033s) [upload=329915]: Done starting upload
Sep 22 04:48:51.827860204 (375h20m27.867433334s) [upload=329915]: Uploading segments...
Sep 22 04:48:51.828124368 (375h20m27.867697488s) [upload=329915]: Got next segment. Inline: false
Sep 22 04:48:51.828212004 (375h20m27.867785124s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:48:51.828220991 (375h20m27.867794121s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:48:52.083032281 (375h20m28.122605401s) [seg_pos={0 0}, upload=329915]: Begin upload segment...
Sep 22 04:48:52.083079671 (375h20m28.122652791s) [seg_pos={0 0}, upload=329915]: Done begin upload segment.
Sep 22 04:48:52.083103035 (375h20m28.122676165s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083129847 (375h20m28.122702967s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083655047 (375h20m28.123228167s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084399634 (375h20m28.123972754s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084672223 (375h20m28.124245353s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084759530 (375h20m28.12433265s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084998324 (375h20m28.124571454s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085115347 (375h20m28.124688467s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085313976 (375h20m28.124887096s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085532622 (375h20m28.125105742s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085983340 (375h20m28.12555646s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086482561 (375h20m28.126055681s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086954470 (375h20m28.12652761s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087292853 (375h20m28.126865983s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087644342 (375h20m28.127217472s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.088021100 (375h20m28.12759422s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:53.832184080 (375h20m29.8717572s) [upload=329915]: Next returned nil segment
Sep 22 04:48:53.832208627 (375h20m29.871781757s) [upload=329915]: Waiting for error group managing segments...
Sep 22 04:48:54.111138623 (375h20m30.150711753s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.174818501 (375h20m30.214391622s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.183976781 (375h20m30.223549901s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.195801608 (375h20m30.235374728s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.205120883 (375h20m30.244694013s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.208206349 (375h20m30.247779479s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.223399449 (375h20m30.262972579s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.243643157 (375h20m30.283216277s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.263265141 (375h20m30.302838271s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.287740006 (375h20m30.327313136s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.319415021 (375h20m30.358988151s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.325619996 (375h20m30.365193117s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:49:21.116551863 (375h20m57.156124983s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116629871 (375h20m57.156203001s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116644179 (375h20m57.156217309s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116719182 (375h20m57.156292312s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116757505 (375h20m57.156330625s) [seg_pos={0 0}, upload=329915]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:49:21.360898295 (375h20m57.400471436s) [upload=329915]: Done uploading segments...
Sep 22 04:49:21.360932911 (375h20m57.400506041s) [upload=329915]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.108948817s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [....XX........X.....X...X......X.X....X.....X....X.....X.X..] duration:
6.299996692s
FAIL (2025-09-22 04:50:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:48:51.827659682 (375h20m27.867232812s) [upload=329915]: Starting upload
Sep 22 04:48:51.827743903 (375h20m27.867317033s) [upload=329915]: Done starting upload
Sep 22 04:48:51.827860204 (375h20m27.867433334s) [upload=329915]: Uploading segments...
Sep 22 04:48:51.828124368 (375h20m27.867697488s) [upload=329915]: Got next segment. Inline: false
Sep 22 04:48:51.828212004 (375h20m27.867785124s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:48:51.828220991 (375h20m27.867794121s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:48:52.083032281 (375h20m28.122605401s) [seg_pos={0 0}, upload=329915]: Begin upload segment...
Sep 22 04:48:52.083079671 (375h20m28.122652791s) [seg_pos={0 0}, upload=329915]: Done begin upload segment.
Sep 22 04:48:52.083103035 (375h20m28.122676165s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083129847 (375h20m28.122702967s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083655047 (375h20m28.123228167s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084399634 (375h20m28.123972754s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084672223 (375h20m28.124245353s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084759530 (375h20m28.12433265s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084998324 (375h20m28.124571454s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085115347 (375h20m28.124688467s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085313976 (375h20m28.124887096s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085532622 (375h20m28.125105742s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085983340 (375h20m28.12555646s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086482561 (375h20m28.126055681s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086954470 (375h20m28.12652761s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087292853 (375h20m28.126865983s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087644342 (375h20m28.127217472s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.088021100 (375h20m28.12759422s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:53.832184080 (375h20m29.8717572s) [upload=329915]: Next returned nil segment
Sep 22 04:48:53.832208627 (375h20m29.871781757s) [upload=329915]: Waiting for error group managing segments...
Sep 22 04:48:54.111138623 (375h20m30.150711753s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.174818501 (375h20m30.214391622s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.183976781 (375h20m30.223549901s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.195801608 (375h20m30.235374728s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.205120883 (375h20m30.244694013s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.208206349 (375h20m30.247779479s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.223399449 (375h20m30.262972579s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.243643157 (375h20m30.283216277s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.263265141 (375h20m30.302838271s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.287740006 (375h20m30.327313136s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.319415021 (375h20m30.358988151s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.325619996 (375h20m30.365193117s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:49:21.116551863 (375h20m57.156124983s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116629871 (375h20m57.156203001s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116644179 (375h20m57.156217309s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116719182 (375h20m57.156292312s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116757505 (375h20m57.156330625s) [seg_pos={0 0}, upload=329915]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:49:21.360898295 (375h20m57.400471436s) [upload=329915]: Done uploading segments...
Sep 22 04:49:21.360932911 (375h20m57.400506041s) [upload=329915]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.300030267s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X......XXXXXXX.X......X.X.X..X.......X.XX.X........XX......] duration:
5.985076899s
FAIL (2025-09-22 04:50:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.985106216s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X......XXXXXXX.X......X.X.X..X.......X.XX.X........XX.....] duration:
6.035794645s
FAIL (2025-09-22 04:50:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.035838147s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X....XX........X.....X...X......X.X....X.....X....X.....X.X.] duration:
6.075652714s
FAIL (2025-09-22 04:49:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:48:51.827659682 (375h20m27.867232812s) [upload=329915]: Starting upload
Sep 22 04:48:51.827743903 (375h20m27.867317033s) [upload=329915]: Done starting upload
Sep 22 04:48:51.827860204 (375h20m27.867433334s) [upload=329915]: Uploading segments...
Sep 22 04:48:51.828124368 (375h20m27.867697488s) [upload=329915]: Got next segment. Inline: false
Sep 22 04:48:51.828212004 (375h20m27.867785124s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:48:51.828220991 (375h20m27.867794121s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:48:52.083032281 (375h20m28.122605401s) [seg_pos={0 0}, upload=329915]: Begin upload segment...
Sep 22 04:48:52.083079671 (375h20m28.122652791s) [seg_pos={0 0}, upload=329915]: Done begin upload segment.
Sep 22 04:48:52.083103035 (375h20m28.122676165s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083129847 (375h20m28.122702967s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083655047 (375h20m28.123228167s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084399634 (375h20m28.123972754s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084672223 (375h20m28.124245353s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084759530 (375h20m28.12433265s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084998324 (375h20m28.124571454s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085115347 (375h20m28.124688467s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085313976 (375h20m28.124887096s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085532622 (375h20m28.125105742s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085983340 (375h20m28.12555646s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086482561 (375h20m28.126055681s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086954470 (375h20m28.12652761s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087292853 (375h20m28.126865983s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087644342 (375h20m28.127217472s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.088021100 (375h20m28.12759422s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:53.832184080 (375h20m29.8717572s) [upload=329915]: Next returned nil segment
Sep 22 04:48:53.832208627 (375h20m29.871781757s) [upload=329915]: Waiting for error group managing segments...
Sep 22 04:48:54.111138623 (375h20m30.150711753s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.174818501 (375h20m30.214391622s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.183976781 (375h20m30.223549901s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.195801608 (375h20m30.235374728s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.205120883 (375h20m30.244694013s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.208206349 (375h20m30.247779479s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.223399449 (375h20m30.262972579s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.243643157 (375h20m30.283216277s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.263265141 (375h20m30.302838271s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.287740006 (375h20m30.327313136s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.319415021 (375h20m30.358988151s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.325619996 (375h20m30.365193117s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:49:21.116551863 (375h20m57.156124983s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116629871 (375h20m57.156203001s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116644179 (375h20m57.156217309s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116719182 (375h20m57.156292312s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116757505 (375h20m57.156330625s) [seg_pos={0 0}, upload=329915]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:49:21.360898295 (375h20m57.400471436s) [upload=329915]: Done uploading segments...
Sep 22 04:49:21.360932911 (375h20m57.400506041s) [upload=329915]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.075688293s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.057325279s
FAIL (2025-09-22 04:49:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload
Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload
Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments...
Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false
Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment...
Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment.
Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment
Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments...
Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments...
Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 6.05735731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X....XX........X.....X...X......X.X....X.....X....X.....X.X] duration:
30.246423634s
FAIL (2025-09-22 04:49:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 22 04:48:51.827659682 (375h20m27.867232812s) [upload=329915]: Starting upload
Sep 22 04:48:51.827743903 (375h20m27.867317033s) [upload=329915]: Done starting upload
Sep 22 04:48:51.827860204 (375h20m27.867433334s) [upload=329915]: Uploading segments...
Sep 22 04:48:51.828124368 (375h20m27.867697488s) [upload=329915]: Got next segment. Inline: false
Sep 22 04:48:51.828212004 (375h20m27.867785124s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:48:51.828220991 (375h20m27.867794121s) [upload=329915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:48:52.083032281 (375h20m28.122605401s) [seg_pos={0 0}, upload=329915]: Begin upload segment...
Sep 22 04:48:52.083079671 (375h20m28.122652791s) [seg_pos={0 0}, upload=329915]: Done begin upload segment.
Sep 22 04:48:52.083103035 (375h20m28.122676165s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083129847 (375h20m28.122702967s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.083655047 (375h20m28.123228167s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084399634 (375h20m28.123972754s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084672223 (375h20m28.124245353s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084759530 (375h20m28.12433265s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.084998324 (375h20m28.124571454s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085115347 (375h20m28.124688467s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085313976 (375h20m28.124887096s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085532622 (375h20m28.125105742s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.085983340 (375h20m28.12555646s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086482561 (375h20m28.126055681s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.086954470 (375h20m28.12652761s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087292853 (375h20m28.126865983s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.087644342 (375h20m28.127217472s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:52.088021100 (375h20m28.12759422s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Uploading piece...
Sep 22 04:48:53.832184080 (375h20m29.8717572s) [upload=329915]: Next returned nil segment
Sep 22 04:48:53.832208627 (375h20m29.871781757s) [upload=329915]: Waiting for error group managing segments...
Sep 22 04:48:54.111138623 (375h20m30.150711753s) [noise=true , address=45.91.116.7:4010 , piece_id=6J3YYQTB4QDJL2B5346XUTOPL6NYTWJDRRX2AWRTDSX2UTQ7WQ2A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.174818501 (375h20m30.214391622s) [noise=true , address=45.91.116.12:4225 , piece_id=BIJRAYHUT6FMOUA2I6WNVXB7NSZT7XYC5MK7BJXHGOG4FNFKWXLA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.183976781 (375h20m30.223549901s) [noise=true , address=77.90.31.19:4843 , piece_id=HTY33RYS5XZJ7TWRXIAD2WBXUMBFNHUYTLMROXZVFATBCMJ7ST4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.195801608 (375h20m30.235374728s) [noise=true , address=45.91.116.8:4047 , piece_id=ZTPNBNXQIPVLKTRQXWMFZPMV7WKOT2XR7QDIXNHPCYGACP6OSOCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.205120883 (375h20m30.244694013s) [noise=true , address=45.91.116.5:4298 , piece_id=TOPDGA43TA3SRIZULROE4JOKF73INIEGPL3IC7YRC6IBO7U52ILQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.208206349 (375h20m30.247779479s) [noise=true , address=45.91.116.7:4011 , piece_id=LPJY6IKQUX6XJS2XNEN2J5JBKW7FYHQZIXNW53JDLLVGAZVZH7IA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.223399449 (375h20m30.262972579s) [noise=true , address=77.90.31.10:4518 , piece_id=CBXUEFE55XQQMPJNM36TCINCK2Q54YYVSEIVFFWVKG5UIUK2HRFA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.243643157 (375h20m30.283216277s) [noise=true , address=45.91.116.6:4975 , piece_id=LSZ57WCLH34PFDXWIIL2BAAFWT2TH7X5J4HR24C36JMDRHR2FHQA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.263265141 (375h20m30.302838271s) [noise=true , address=77.90.31.15:4698 , piece_id=KHXNRTSPVTPK4MHXIZ4NWPVUTPY4RDY3JA36A4CX2AQKMEPU5CCQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.287740006 (375h20m30.327313136s) [noise=true , address=77.90.31.16:4733 , piece_id=6T2JS4766NECWN745EEB5MHNLEJEVGSUKUCS42MK5HEFUKKIBNAQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.319415021 (375h20m30.358988151s) [noise=true , address=45.91.116.4:4938 , piece_id=4AFAM3QTFETL5S3GLC25IKWQN3ENC56JRVFYK5CZCKNCXTDEMG5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:48:54.325619996 (375h20m30.365193117s) [noise=true , address=45.91.116.9:4261 , piece_id=HA377ZFMSKU5YRGQWGELUAFCILW4B2HWNCBSNUEMNNSNUTSQSVZQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: <nil>
Sep 22 04:49:21.116551863 (375h20m57.156124983s) [noise=true , address=145.79.227.15:4875 , piece_id=FEBYQETO4JGLMA5JBSBSZDFLCXJICYOJQHKWQED2IASNJBORVWEA, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116629871 (375h20m57.156203001s) [noise=true , address=145.79.227.11:4729 , piece_id=T3CXFY44OJ76PNSGLYZ4DBG7B4KPUERMZXDI2L5NIKFPHGZKFC5A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116644179 (375h20m57.156217309s) [noise=true , address=145.79.227.18:4007 , piece_id=NKTQXJQHF6XNQLJ5RUCQLAQGE4YVMGW3IHDYSB6UW3P7BKR3AI4A, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116719182 (375h20m57.156292312s) [noise=true , address=145.79.227.12:4767 , piece_id=MKQSC2VIFKZ5M62JU4XLZGJ4PDPIZWNXUSRGJDTWMBQKWPIAGUSQ, seg_pos={0 0}, upload=329915]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:49:21.116757505 (375h20m57.156330625s) [seg_pos={0 0}, upload=329915]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 22 04:49:21.360898295 (375h20m57.400471436s) [upload=329915]: Done uploading segments...
Sep 22 04:49:21.360932911 (375h20m57.400506041s) [upload=329915]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.246441519s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
6.086464803s
FAIL (2025-09-22 04:49:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
Sep 22 04:05:29.619615389 (374h37m5.659188519s) [upload=329273]: Starting upload
Sep 22 04:05:29.619743261 (374h37m5.659316391s) [upload=329273]: Done starting upload
Sep 22 04:05:29.619869021 (374h37m5.659442151s) [upload=329273]: Uploading segments...
Sep 22 04:05:29.620023575 (374h37m5.659596705s) [upload=329273]: Got next segment. Inline: false
Sep 22 04:05:29.620076637 (374h37m5.659649757s) [upload=329273]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 22 04:05:29.620083369 (374h37m5.659656489s) [upload=329273]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 22 04:05:30.206830039 (374h37m6.246403169s) [seg_pos={0 0}, upload=329273]: Begin upload segment...
Sep 22 04:05:30.207012397 (374h37m6.246585527s) [seg_pos={0 0}, upload=329273]: Done begin upload segment.
Sep 22 04:05:30.207057543 (374h37m6.246630663s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.208427230 (374h37m6.24800037s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.209291286 (374h37m6.248864416s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.209882782 (374h37m6.249455922s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210249010 (374h37m6.24982214s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210571053 (374h37m6.250144183s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210688977 (374h37m6.250262107s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.210921490 (374h37m6.25049462s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211200461 (374h37m6.250773581s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211280263 (374h37m6.250853393s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211634527 (374h37m6.251207657s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.211982650 (374h37m6.25155578s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.212352886 (374h37m6.251926016s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.212714916 (374h37m6.252288046s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.213077496 (374h37m6.252650626s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.213629317 (374h37m6.253202447s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.214015542 (374h37m6.253588662s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:30.214362753 (374h37m6.253935883s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Uploading piece...
Sep 22 04:05:31.669230775 (374h37m7.708803895s) [upload=329273]: Next returned nil segment
Sep 22 04:05:31.669258689 (374h37m7.708831839s) [upload=329273]: Waiting for error group managing segments...
Sep 22 04:05:32.037129875 (374h37m8.076703005s) [noise=true , address=64.49.13.169:4706 , piece_id=DNUOX2DEIOCNQPXH56TSUSENKKZ5WPXSC2HUOL7QSKE2FWMV3VMQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.094107712 (374h37m8.133680842s) [noise=true , address=145.79.227.25:4625 , piece_id=SQD3WH3GKA7XHKQWGEVPKU5RODRBDOLBDU44QC3HMIH5JXYDL3NA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.101606951 (374h37m8.141180081s) [noise=true , address=145.79.227.27:4718 , piece_id=X5D5MR5VJ7PVHU4DMC3AK6QRCSR2TGNAQJA3TE5FSA7Q7L2WHJXQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.101887065 (374h37m8.141460195s) [noise=true , address=145.79.227.39:4036 , piece_id=3KNUJENKWYC47XKTPNSAXKJI7TDJRBQEL3CZBEYF62LI73WJQPVA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.104942563 (374h37m8.144515693s) [noise=true , address=145.79.227.29:4960 , piece_id=XRWCNNBAMUTMFAUHY6UIHYH65HN4T36ZF2423CKPWO7V6R7R6OTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.106424105 (374h37m8.145997225s) [noise=true , address=145.79.227.33:4904 , piece_id=QVYEDEFYTVOL7OZDYGEY3B3HDAQUIIB4FI57QSMHGA7H3ODEUBOQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.109198217 (374h37m8.148771347s) [noise=true , address=145.79.227.22:4544 , piece_id=IYY5HXWRBPTQBZH3DXXKSMSLTA6CJYOKFAFX7BAZHAHBUQS7MTRQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.109775015 (374h37m8.149348145s) [noise=true , address=145.79.227.48:4302 , piece_id=72IRQJT5UYCZB7PWDW6WBQAULFAQFOGCINWYHJACNOBRXLSHVCNA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.111164401 (374h37m8.150737531s) [noise=true , address=145.79.227.35:4499 , piece_id=J5OCXMHPFA7DHVOLSJK3W5QDVVBUQZJQQBMOP7PVMUJIDTFG3T5Q, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.111711984 (374h37m8.151285114s) [noise=true , address=145.79.227.46:4619 , piece_id=S4CRF5ZN44BNW2Z3TALPCBJSX3Z3JH7VBQV2K2GXZ6ADJTVVPKAA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.112386858 (374h37m8.151959989s) [noise=true , address=145.79.227.26:4655 , piece_id=MA2CYBV5KJDCZW2OOORPXOCTHY6AY7MS3CBFTAPHTYEZQQMOADHQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.112911798 (374h37m8.152484928s) [noise=true , address=145.79.227.21:4475 , piece_id=QSV7BAAVKYNIM4AOXPKEKPK24H5JTRCRY2SWT3NYLB3HS6T5AHBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:05:32.144364670 (374h37m8.18393779s) [noise=true , address=145.79.227.31:4813 , piece_id=7LLRCMC65H4OXWGRWACRRYB4NGAMKEB2ADBJ25PKTUYHF7XZWCBA, seg_pos={0 0}, upload=329273]: Done uploading piece. err: <nil>
Sep 22 04:06:28.869364778 (374h38m4.908937938s) [noise=true , address=145.79.227.11:4750 , piece_id=DPU4W6HAV6FSS6R4JGYHEFPV242VVQMD3KLJNO55YLNYOYCLURTQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869503032 (374h38m4.909076162s) [noise=true , address=145.79.227.12:4773 , piece_id=AMLZ3UENZZI7QBK2CZZTIGQSSHUMK6IPDLV6E35XXJJC2JDDG2ZQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:145.79.227.12:4773): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869542396 (374h38m4.909115527s) [noise=true , address=145.79.227.13:4831 , piece_id=X4QUOWFRAA4GC2YPSXZ64LUO6TLPASKJEYAWOFJCJNUIZNQZNX6A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:145.79.227.13:4831): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869592672 (374h38m4.909165792s) [noise=true , address=145.79.227.2:4131 , piece_id=DXQ2TKGZA5PQODJIJRS5LVA5WAFFGU44WHW4P2BD32Z3GEFH2KQQ, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:145.79.227.2:4131): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869693524 (374h38m4.909266654s) [noise=true , address=145.79.227.18:4031 , piece_id=KXEE5MTPZUEAP6YECGKFLWLFF2VD5ADORWRM6JUSC7FJNLDGG73A, seg_pos={0 0}, upload=329273]: Done uploading piece. err: ecclient: upload failed (node:12Hx9KabcKhsb2oGPefNr12E2ibDkaVnPJ6Q38rtxxudkDdXMta, address:145.79.227.18:4031): protocol: context deadline exceeded; context deadline exceeded
Sep 22 04:06:28.869721798 (374h38m4.909294928s) [seg_pos={0 0}, upload=329273]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Sep 22 04:06:29.123952782 (374h38m5.163525902s) [upload=329273]: Done uploading segments...
Sep 22 04:06:29.123990093 (374h38m5.163563213s) [upload=329273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 6.086500412s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X......XXXXXXX.X......X.X.X..X.......X.XX.X........XX....] duration:
6.079717578s
FAIL (2025-09-22 04:49:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.079989224s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X......XXXXXXX.X......X.X.X..X.......X.XX.X........XX...] duration:
6.154208754s
FAIL (2025-09-22 04:48:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.154241246s
edge
gateway
us-select-1
|