Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........XX........X...................X.....X............] duration:
6.372966421s
FAIL (2025-10-06 23:29:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.372986128s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.828989756s
FAIL (2025-10-06 23:29:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.82901272s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X...........XX.......X..X..X...X....X...............] duration:
6.208461886s
FAIL (2025-10-06 23:29:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.208493586s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............XX........X...................X.....X...........] duration:
6.230157249s
FAIL (2025-10-06 23:28:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.230190702s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X..................X......X..X....................] duration:
6.087144718s
FAIL (2025-10-06 23:28:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.087167941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.732828147s
FAIL (2025-10-06 23:28:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.732858435s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.........X...........XX.......X..X..X...X....X..............] duration:
5.867088807s
FAIL (2025-10-06 23:28:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.867124494s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X..................X......X..X...................] duration:
6.032819803s
FAIL (2025-10-06 23:28:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.032852826s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............XX........X...................X.....X..........] duration:
6.480192899s
FAIL (2025-10-06 23:27:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.480217184s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............XX........X...................X.....X.........] duration:
6.218376012s
FAIL (2025-10-06 23:27:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.218401853s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..........X...........XX.......X..X..X...X....X.............] duration:
6.330300527s
FAIL (2025-10-06 23:27:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.330335293s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.685489486s
FAIL (2025-10-06 23:27:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.685540103s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X..................X......X..X..................] duration:
5.903804414s
FAIL (2025-10-06 23:27:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.903856384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........X...........XX.......X..X..X...X....X............] duration:
5.931642415s
FAIL (2025-10-06 23:26:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.931680127s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.71905414s
FAIL (2025-10-06 23:26:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.719156795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............XX........X...................X.....X........] duration:
6.233211593s
FAIL (2025-10-06 23:26:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.233255906s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X..................X......X..X.................] duration:
6.126011756s
FAIL (2025-10-06 23:26:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.126038256s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.816263022s
FAIL (2025-10-06 23:26:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.816319349s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [............X...........XX.......X..X..X...X....X...........] duration:
6.194070149s
FAIL (2025-10-06 23:26:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.194100087s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X..................X......X..X................] duration:
5.909915752s
FAIL (2025-10-06 23:25:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.909945969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.608413411s
FAIL (2025-10-06 23:25:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.608438498s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............X...........XX.......X..X..X...X....X..........] duration:
6.313772444s
FAIL (2025-10-06 23:25:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.313805067s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................XX........X...................X.....X.......] duration:
6.101147019s
FAIL (2025-10-06 23:25:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.10117325s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............X...........XX.......X..X..X...X....X.........] duration:
6.346444088s
FAIL (2025-10-06 23:25:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.346477152s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
5.822275039s
FAIL (2025-10-06 23:25:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.822373076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X..................X......X..X...............] duration:
6.23794703s
FAIL (2025-10-06 23:25:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.237971587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................XX........X...................X.....X......] duration:
6.307661159s
FAIL (2025-10-06 23:24:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.308482312s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X..................X......X..X..............] duration:
5.952428664s
FAIL (2025-10-06 23:24:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.952458931s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...............X...........XX.......X..X..X...X....X........] duration:
6.070092827s
FAIL (2025-10-06 23:24:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.070124297s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X..................X......X..X.............] duration:
6.005290843s
FAIL (2025-10-06 23:24:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.005319636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.560532739s
FAIL (2025-10-06 23:24:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.56064325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................XX........X...................X.....X.....] duration:
6.139511902s
FAIL (2025-10-06 23:24:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.139537581s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X..................X......X..X............] duration:
6.001865444s
FAIL (2025-10-06 23:23:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.001901053s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [................X...........XX.......X..X..X...X....X.......] duration:
6.180760286s
FAIL (2025-10-06 23:23:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.180786595s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
5.746938528s
FAIL (2025-10-06 23:23:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.746965898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................X..................X......X..X...........] duration:
5.95629145s
FAIL (2025-10-06 23:23:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.956324433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
5.675253255s
FAIL (2025-10-06 23:23:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.675279895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.................X...........XX.......X..X..X...X....X......] duration:
6.22565869s
FAIL (2025-10-06 23:22:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.225687876s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................XX........X...................X.....X....] duration:
6.286882913s
FAIL (2025-10-06 23:22:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.286900205s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................X..................X......X..X..........] duration:
5.995936952s
FAIL (2025-10-06 23:22:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.995968141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..................X...........XX.......X..X..X...X....X.....] duration:
5.893074283s
FAIL (2025-10-06 23:22:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 5.893103319s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X..................X......X..X.........] duration:
6.045390525s
FAIL (2025-10-06 23:22:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.045426202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.697448682s
FAIL (2025-10-06 23:21:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.697474742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................XX........X...................X.....X...] duration:
6.351454694s
FAIL (2025-10-06 23:21:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.351476556s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...................X...........XX.......X..X..X...X....X....] duration:
6.303100481s
FAIL (2025-10-06 23:21:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.3031268s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X..................X......X..X........] duration:
7.052384839s
FAIL (2025-10-06 23:21:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
ch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment...
Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment.
Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece...
Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment
Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments...
Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil>
Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments...
Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 7.052415507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.834915965s
FAIL (2025-10-06 23:21:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.834940852s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....................X...........XX.......X..X..X...X....X...] duration:
6.383159486s
FAIL (2025-10-06 23:20:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.383191767s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................XX........X...................X.....X..] duration:
6.682696606s
FAIL (2025-10-06 23:20:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.682724208s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....................X...........XX.......X..X..X...X....X..] duration:
6.897868073s
FAIL (2025-10-06 23:20:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload
Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload
Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments...
Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false
Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment...
Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment.
Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece...
Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment
Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments...
Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil>
Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded
Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments...
Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 6.897901716s
core
uplink
us-select-1
|