Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [........XX.......X..X..X...X....X...........................] duration:
6.220902916s
FAIL (2025-10-06 23:37:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.220934766s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX........X...................X.....X.......................] duration:
6.509038945s
FAIL (2025-10-06 23:37:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.509070836s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X......X..X.................................] duration:
5.791892525s
FAIL (2025-10-06 23:37:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.791928906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.60653304s
FAIL (2025-10-06 23:36:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.606610367s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X......X..X................................] duration:
5.737334779s
FAIL (2025-10-06 23:36:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.737365768s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX........X...................X.....X......................] duration:
6.209097806s
FAIL (2025-10-06 23:36:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.209126219s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.........XX.......X..X..X...X....X..........................] duration:
5.813248711s
FAIL (2025-10-06 23:36:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.81328484s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X......X..X...............................] duration:
6.043545055s
FAIL (2025-10-06 23:36:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.043574663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX........X...................X.....X.....................] duration:
6.241348939s
FAIL (2025-10-06 23:36:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.2413812s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..........XX.......X..X..X...X....X.........................] duration:
5.874232044s
FAIL (2025-10-06 23:36:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.874325843s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.670070515s
FAIL (2025-10-06 23:35:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.670095251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...........XX.......X..X..X...X....X........................] duration:
6.154554405s
FAIL (2025-10-06 23:35:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.154583551s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..................X......X..X..............................] duration:
5.888916106s
FAIL (2025-10-06 23:35:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.888948137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XX........X...................X.....X....................] duration:
6.431239583s
FAIL (2025-10-06 23:35:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.431265641s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.674975356s
FAIL (2025-10-06 23:34:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.675000645s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [X...........XX.......X..X..X...X....X.......................] duration:
6.166231301s
FAIL (2025-10-06 23:34:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.16626251s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..................X......X..X.............................] duration:
5.834671937s
FAIL (2025-10-06 23:34:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.834701933s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X...........XX.......X..X..X...X....X......................] duration:
6.250054123s
FAIL (2025-10-06 23:34:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.25008422s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XX........X...................X.....X...................] duration:
6.596808524s
FAIL (2025-10-06 23:34:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.596833803s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..................X......X..X............................] duration:
6.06074647s
FAIL (2025-10-06 23:34:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.060778831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.598289833s
FAIL (2025-10-06 23:34:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.598327175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..................X......X..X...........................] duration:
6.041296215s
FAIL (2025-10-06 23:33:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.041329488s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.819667244s
FAIL (2025-10-06 23:33:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.819692312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X...........XX.......X..X..X...X....X.....................] duration:
6.546433672s
FAIL (2025-10-06 23:33:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.546466925s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....XX........X...................X.....X..................] duration:
6.292713335s
FAIL (2025-10-06 23:33:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.292739423s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......XX........X...................X.....X.................] duration:
6.186925308s
FAIL (2025-10-06 23:32:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.186950456s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..................X......X..X..........................] duration:
5.820219626s
FAIL (2025-10-06 23:32:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.82024802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.666558088s
FAIL (2025-10-06 23:32:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.666583506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X...........XX.......X..X..X...X....X....................] duration:
6.060990886s
FAIL (2025-10-06 23:32:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.061026163s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......XX........X...................X.....X................] duration:
6.090835206s
FAIL (2025-10-06 23:32:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.090867146s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..................X......X..X.........................] duration:
5.972089724s
FAIL (2025-10-06 23:31:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.972162534s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....X...........XX.......X..X..X...X....X...................] duration:
5.996705273s
FAIL (2025-10-06 23:31:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.996736011s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
5.750646435s
FAIL (2025-10-06 23:31:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.750670511s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X..................X......X..X........................] duration:
5.935868079s
FAIL (2025-10-06 23:31:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.935901152s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........XX........X...................X.....X...............] duration:
6.268272522s
FAIL (2025-10-06 23:31:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.268301678s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.868389831s
FAIL (2025-10-06 23:30:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.868407383s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X..................X......X..X.......................] duration:
5.932511739s
FAIL (2025-10-06 23:30:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.932556013s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.792341019s
FAIL (2025-10-06 23:30:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.792378901s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X...........XX.......X..X..X...X....X..................] duration:
6.201457378s
FAIL (2025-10-06 23:30:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.201487215s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........XX........X...................X.....X..............] duration:
6.168219739s
FAIL (2025-10-06 23:30:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.168253623s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X..................X......X..X......................] duration:
6.216015328s
FAIL (2025-10-06 23:30:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.216056647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.728616898s
FAIL (2025-10-06 23:29:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.728671542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......X...........XX.......X..X..X...X....X.................] duration:
6.053137542s
FAIL (2025-10-06 23:29:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.053165234s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X...........XX.......X..X..X...X....X................] duration:
6.381498326s
FAIL (2025-10-06 23:29:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.381532943s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.625251294s
FAIL (2025-10-06 23:29:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.625271063s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........XX........X...................X.....X.............] duration:
6.082260836s
FAIL (2025-10-06 23:29:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.082289551s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X..................X......X..X.....................] duration:
5.983072676s
FAIL (2025-10-06 23:29:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.983102373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........XX........X...................X.....X............] duration:
6.372966421s
FAIL (2025-10-06 23:29:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.372986128s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.828989756s
FAIL (2025-10-06 23:29:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.82901272s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X...........XX.......X..X..X...X....X...............] duration:
6.208461886s
FAIL (2025-10-06 23:29:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.208493586s
core
uplink
us-select-1
|