Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration:
6.399341238s
FAIL (2025-10-06 23:46:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.399395602s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration:
6.243618023s
FAIL (2025-10-06 23:46:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.243643221s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.888155281s
FAIL (2025-10-06 23:46:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.888208062s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X......X..X.............................................] duration:
6.002858766s
FAIL (2025-10-06 23:45:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.002922877s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X..X..X...X....X........................................] duration:
6.216466701s
FAIL (2025-10-06 23:45:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.216496096s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X.....X..................................] duration:
6.167482886s
FAIL (2025-10-06 23:45:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.167509167s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.555250502s
FAIL (2025-10-06 23:45:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.555283997s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X..X..X...X....X.......................................] duration:
6.407275664s
FAIL (2025-10-06 23:44:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.407307124s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X......X..X............................................] duration:
6.058319432s
FAIL (2025-10-06 23:44:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.05834978s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X...................X.....X.................................] duration:
6.397015547s
FAIL (2025-10-06 23:44:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.397046387s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X..X..X...X....X......................................] duration:
6.495083325s
FAIL (2025-10-06 23:44:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.495135775s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.80268299s
FAIL (2025-10-06 23:44:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.802712887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X...................X.....X................................] duration:
6.265880344s
FAIL (2025-10-06 23:44:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.265905583s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X......X..X...........................................] duration:
5.860995674s
FAIL (2025-10-06 23:44:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.861033026s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X...................X.....X...............................] duration:
6.080905016s
FAIL (2025-10-06 23:43:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.08093295s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X..X..X...X....X.....................................] duration:
6.214892264s
FAIL (2025-10-06 23:43:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.214925277s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.665409605s
FAIL (2025-10-06 23:43:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.665450974s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X......X..X..........................................] duration:
5.862588467s
FAIL (2025-10-06 23:43:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.862633153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X.......X..X..X...X....X....................................] duration:
6.118073123s
FAIL (2025-10-06 23:43:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.118105085s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [XX.......X..X..X...X....X...................................] duration:
6.180204734s
FAIL (2025-10-06 23:42:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.180236322s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X...................X.....X..............................] duration:
6.525098199s
FAIL (2025-10-06 23:42:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.525125783s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.XX.......X..X..X...X....X..................................] duration:
6.094010016s
FAIL (2025-10-06 23:42:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.094040995s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.8388314s
FAIL (2025-10-06 23:42:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.838858993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X......X..X.........................................] duration:
6.196114508s
FAIL (2025-10-06 23:42:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.196242731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X...................X.....X.............................] duration:
6.463450031s
FAIL (2025-10-06 23:42:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.463478585s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..XX.......X..X..X...X....X.................................] duration:
6.186263831s
FAIL (2025-10-06 23:41:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.186292475s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X......X..X........................................] duration:
5.832316043s
FAIL (2025-10-06 23:41:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.832359906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.744042487s
FAIL (2025-10-06 23:41:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.744068867s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...XX.......X..X..X...X....X................................] duration:
5.854454058s
FAIL (2025-10-06 23:41:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.854479005s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X...................X.....X............................] duration:
6.438798728s
FAIL (2025-10-06 23:40:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.43882591s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.109531229s
FAIL (2025-10-06 23:40:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.109558751s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X......X..X.......................................] duration:
6.186764454s
FAIL (2025-10-06 23:40:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.186796255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [....XX.......X..X..X...X....X...............................] duration:
5.899113215s
FAIL (2025-10-06 23:40:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.899149775s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X......X..X......................................] duration:
5.9309675s
FAIL (2025-10-06 23:40:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.931003369s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X...................X.....X...........................] duration:
6.602924794s
FAIL (2025-10-06 23:40:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.60302151s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.135764271s
FAIL (2025-10-06 23:40:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.135788177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X......X..X.....................................] duration:
5.812091143s
FAIL (2025-10-06 23:39:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.812126399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.....XX.......X..X..X...X....X..............................] duration:
6.598393932s
FAIL (2025-10-06 23:39:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.598439269s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.722975706s
FAIL (2025-10-06 23:39:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.723005845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X...................X.....X..........................] duration:
6.50126923s
FAIL (2025-10-06 23:39:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.501298727s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X......X..X....................................] duration:
5.9527413s
FAIL (2025-10-06 23:39:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.952775776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.801177927s
FAIL (2025-10-06 23:39:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.801280231s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X...................X.....X.........................] duration:
6.320885098s
FAIL (2025-10-06 23:38:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.320911909s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [......XX.......X..X..X...X....X.............................] duration:
6.065020639s
FAIL (2025-10-06 23:38:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.065041298s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.......XX.......X..X..X...X....X............................] duration:
6.453480473s
FAIL (2025-10-06 23:38:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.453524727s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X......X..X...................................] duration:
6.035853851s
FAIL (2025-10-06 23:38:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.035886183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.638855926s
FAIL (2025-10-06 23:38:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.638887898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X........X...................X.....X........................] duration:
6.216747512s
FAIL (2025-10-06 23:38:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.216778371s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.56343698s
FAIL (2025-10-06 23:37:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.563465694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X......X..X..................................] duration:
6.225899046s
FAIL (2025-10-06 23:37:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.225938062s
core
uplink
us-select-1
us1-placement-20
|