Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X......X..X.......................................] duration:
6.186764454s
FAIL (2025-10-06 23:40:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.186796255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [....XX.......X..X..X...X....X...............................] duration:
5.899113215s
FAIL (2025-10-06 23:40:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.899149775s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X......X..X......................................] duration:
5.9309675s
FAIL (2025-10-06 23:40:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.931003369s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X...................X.....X...........................] duration:
6.602924794s
FAIL (2025-10-06 23:40:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.60302151s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.135764271s
FAIL (2025-10-06 23:40:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.135788177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X......X..X.....................................] duration:
5.812091143s
FAIL (2025-10-06 23:39:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.812126399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....XX.......X..X..X...X....X..............................] duration:
6.598393932s
FAIL (2025-10-06 23:39:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.598439269s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.722975706s
FAIL (2025-10-06 23:39:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.723005845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X...................X.....X..........................] duration:
6.50126923s
FAIL (2025-10-06 23:39:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.501298727s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X......X..X....................................] duration:
5.9527413s
FAIL (2025-10-06 23:39:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.952775776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.801177927s
FAIL (2025-10-06 23:39:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.801280231s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X...................X.....X.........................] duration:
6.320885098s
FAIL (2025-10-06 23:38:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.320911909s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [......XX.......X..X..X...X....X.............................] duration:
6.065020639s
FAIL (2025-10-06 23:38:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.065041298s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.......XX.......X..X..X...X....X............................] duration:
6.453480473s
FAIL (2025-10-06 23:38:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.453524727s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X......X..X...................................] duration:
6.035853851s
FAIL (2025-10-06 23:38:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.035886183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.638855926s
FAIL (2025-10-06 23:38:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.638887898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X........X...................X.....X........................] duration:
6.216747512s
FAIL (2025-10-06 23:38:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.216778371s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.56343698s
FAIL (2025-10-06 23:37:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.563465694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X......X..X..................................] duration:
6.225899046s
FAIL (2025-10-06 23:37:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.225938062s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [........XX.......X..X..X...X....X...........................] duration:
6.220902916s
FAIL (2025-10-06 23:37:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.220934766s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX........X...................X.....X.......................] duration:
6.509038945s
FAIL (2025-10-06 23:37:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.509070836s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X......X..X.................................] duration:
5.791892525s
FAIL (2025-10-06 23:37:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.791928906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.60653304s
FAIL (2025-10-06 23:36:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.606610367s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X......X..X................................] duration:
5.737334779s
FAIL (2025-10-06 23:36:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.737365768s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX........X...................X.....X......................] duration:
6.209097806s
FAIL (2025-10-06 23:36:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.209126219s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.........XX.......X..X..X...X....X..........................] duration:
5.813248711s
FAIL (2025-10-06 23:36:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.81328484s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X......X..X...............................] duration:
6.043545055s
FAIL (2025-10-06 23:36:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.043574663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX........X...................X.....X.....................] duration:
6.241348939s
FAIL (2025-10-06 23:36:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.2413812s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..........XX.......X..X..X...X....X.........................] duration:
5.874232044s
FAIL (2025-10-06 23:36:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.874325843s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.670070515s
FAIL (2025-10-06 23:35:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.670095251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...........XX.......X..X..X...X....X........................] duration:
6.154554405s
FAIL (2025-10-06 23:35:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.154583551s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..................X......X..X..............................] duration:
5.888916106s
FAIL (2025-10-06 23:35:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.888948137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XX........X...................X.....X....................] duration:
6.431239583s
FAIL (2025-10-06 23:35:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.431265641s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.674975356s
FAIL (2025-10-06 23:34:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.675000645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [X...........XX.......X..X..X...X....X.......................] duration:
6.166231301s
FAIL (2025-10-06 23:34:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.16626251s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..................X......X..X.............................] duration:
5.834671937s
FAIL (2025-10-06 23:34:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.834701933s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X...........XX.......X..X..X...X....X......................] duration:
6.250054123s
FAIL (2025-10-06 23:34:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.25008422s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XX........X...................X.....X...................] duration:
6.596808524s
FAIL (2025-10-06 23:34:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.596833803s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..................X......X..X............................] duration:
6.06074647s
FAIL (2025-10-06 23:34:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.060778831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.598289833s
FAIL (2025-10-06 23:34:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.598327175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..................X......X..X...........................] duration:
6.041296215s
FAIL (2025-10-06 23:33:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.041329488s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.819667244s
FAIL (2025-10-06 23:33:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.819692312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X...........XX.......X..X..X...X....X.....................] duration:
6.546433672s
FAIL (2025-10-06 23:33:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.546466925s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XX........X...................X.....X..................] duration:
6.292713335s
FAIL (2025-10-06 23:33:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.292739423s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XX........X...................X.....X.................] duration:
6.186925308s
FAIL (2025-10-06 23:32:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.186950456s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..................X......X..X..........................] duration:
5.820219626s
FAIL (2025-10-06 23:32:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.82024802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.666558088s
FAIL (2025-10-06 23:32:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.666583506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X...........XX.......X..X..X...X....X....................] duration:
6.060990886s
FAIL (2025-10-06 23:32:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.061026163s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XX........X...................X.....X................] duration:
6.090835206s
FAIL (2025-10-06 23:32:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.090867146s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..................X......X..X.........................] duration:
5.972089724s
FAIL (2025-10-06 23:31:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.972162534s
core
uplink
us-select-1
us1-placement-20
|