Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.....XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.........] duration:
30.384339509s
FAIL (2025-07-27 00:06:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.38435079s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.X] duration:
30.23505612s
FAIL (2025-07-27 00:05:52 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.235082952s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X........] duration:
38.482893493s
FAIL (2025-07-27 00:05:33 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.482914954s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XX] duration:
30.241147367s
FAIL (2025-07-27 00:05:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments...
Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:05:02.560967093 (60h29m12.283530133s) [upload=58013]: Starting upload
Jul 27 00:05:02.561059250 (60h29m12.2836223s) [upload=58013]: Done starting upload
Jul 27 00:05:02.561208304 (60h29m12.283771364s) [upload=58013]: Uploading segments...
Jul 27 00:05:02.561391195 (60h29m12.283954245s) [upload=58013]: Got next segment. Inline: false
Jul 27 00:05:02.561445709 (60h29m12.284008749s) [upload=58013]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:05:02.561452602 (60h29m12.284015652s) [upload=58013]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:05:02.798729347 (60h29m12.521292407s) [seg_pos={0 0}, upload=58013]: Begin upload segment...
Jul 27 00:05:02.798805122 (60h29m12.521368162s) [seg_pos={0 0}, upload=58013]: Done begin upload segment.
Jul 27 00:05:02.798838295 (60h29m12.521401346s) [noise=true , address=77.90.31.13:4626 , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.798888512 (60h29m12.521451552s) [noise=true , address=77.90.31.15:4699 , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799021236 (60h29m12.521584276s) [noise=true , address=104.36.82.3:4765 , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799451228 (60h29m12.522014278s) [noise=true , address=45.91.116.10:4081 , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799464424 (60h29m12.522027464s) [noise=true , address=77.90.31.12:4590 , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799794074 (60h29m12.522357124s) [noise=true , address=45.91.116.7:4011 , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.799850472 (60h29m12.522413512s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800176917 (60h29m12.522739967s) [noise=true , address=45.91.116.14:4190 , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800307928 (60h29m12.522870978s) [noise=true , address=77.90.31.14:4663 , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.800695489 (60h29m12.523258529s) [noise=true , address=104.36.82.12:4150 , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801003488 (60h29m12.523566528s) [noise=true , address=77.90.31.10:4518 , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801329863 (60h29m12.523892913s) [noise=true , address=77.90.31.18:4806 , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.801678690 (60h29m12.52424174s) [noise=true , address=104.36.82.11:4113 , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.802064338 (60h29m12.524627388s) [noise=true , address=77.90.31.19:4842 , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.802411622 (60h29m12.524974672s) [noise=true , address=45.91.116.11:4118 , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:02.805333012 (60h29m12.527896052s) [noise=true , address=104.36.82.4:4801 , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Uploading piece...
Jul 27 00:05:04.611541965 (60h29m14.334105005s) [upload=58013]: Next returned nil segment
Jul 27 00:05:04.611564408 (60h29m14.334127448s) [upload=58013]: Waiting for error group managing segments...
Jul 27 00:05:04.914940269 (60h29m14.637503309s) [noise=true , address=45.91.116.10:4081 , piece_id=H42UNKPAPYBC6Y4GKMMX7MMC44LJ4L5IYRHKSZGTFWVN42QMS3RA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.918629126 (60h29m14.641192166s) [noise=true , address=45.91.116.7:4011 , piece_id=BWDT7MTYRODWMXJIA3POFLWAPCTSY4HD4A5KQNVA6KAOAVIIS3ZA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.948737141 (60h29m14.671300181s) [noise=true , address=45.91.116.11:4118 , piece_id=3FV6QW2IXNZASQOCZU6SFIYY3MEFLGQFGQVJZI5ZCS76RJWJGRDA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.963852912 (60h29m14.686415952s) [noise=true , address=45.91.116.13:4154 , piece_id=PLKORLXGDHVD4D25KFP5QW2ZLPREGY3E5TAQBWS23HZVKUO5I3LQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.988745859 (60h29m14.711308929s) [noise=true , address=77.90.31.14:4663 , piece_id=7K3PMS67INYORWMGTQDJ7QI33F562GARO53AHPOI4FUFOC6EUJBQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:04.991103750 (60h29m14.71366679s) [noise=true , address=45.91.116.14:4190 , piece_id=J6QDMYOMJA25SKX2LZE6MAOGMIZGK2SEIHEA6QUW4ONCG5V2PLQA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.248956147 (60h29m15.971519197s) [noise=true , address=77.90.31.18:4806 , piece_id=L6CQHIKXKTDX7HKB3CSWSLAAZDDXU5UD522SHAAHBDVFI5RZ2H6Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.300358975 (60h29m16.022922016s) [noise=true , address=77.90.31.13:4626 , piece_id=2MZ5QDUHJCJZ2CRAJFU3L4RVLIGP32ERMLCR5DAKADWU5MKUQJAA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.410168350 (60h29m16.13273139s) [noise=true , address=77.90.31.12:4590 , piece_id=HTERKMF2LZG23VQ3AMBORXNVYOFC65OKVYVKICO5BXP23HZ4LOFA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:06.869083350 (60h29m16.59164639s) [noise=true , address=77.90.31.19:4842 , piece_id=4NVG57UEGSRZD3LJFIAIN25YACV6336TARYWSGMDE6FYX4FNZBRQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: <nil>
Jul 27 00:05:31.913473966 (60h29m41.636036996s) [noise=true , address=104.36.82.4:4801 , piece_id=UPUBUUVGCNYNH7QPFL37X4PAEDIM443S63K2AUV6PFFUKSSLX4QA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913502281 (60h29m41.636065321s) [noise=true , address=77.90.31.10:4518 , piece_id=JBD56FUTX423KXWYUXY4MZ7B3GY6LUMBDQACUR35JJ47X7U2EA7A, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: read tcp 188.166.225.170:43888->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913521568 (60h29m41.636084608s) [noise=true , address=104.36.82.3:4765 , piece_id=HBMIVDDLHPBH3DCBKB4CU4QY5O47FWRKDDR6OKAGCBAYRRT55GVA, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:56046->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913530945 (60h29m41.636093985s) [noise=true , address=77.90.31.15:4699 , piece_id=S3T2ORLRPFLW6ISGDGSHR7U6RPPQIUU2A3EO5AE7SWHFBLCTAX4Q, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913536205 (60h29m41.636099245s) [noise=true , address=104.36.82.12:4150 , piece_id=IQU5VES2L645KCAZFG63GUO6F5SEBQPQM4OH6A2RPRWVJFYVO2UQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913552858 (60h29m41.636115898s) [noise=true , address=104.36.82.11:4113 , piece_id=ZGQHVEQGXPLVS3HACWUOSXCLBGSBCRYQ7HIU5PXWJE4WSSYKCCJQ, seg_pos={0 0}, upload=58013]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:31.913571243 (60h29m41.636134283s) [seg_pos={0 0}, upload=58013]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:05:32.152667560 (60h29m41.8752306s) [upload=58013]: Done uploading segments...
Jul 27 00:05:32.152698479 (60h29m41.87526152s) [upload=58013]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.241166312s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.65925967s
FAIL (2025-07-27 00:05:10 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 2.659289076s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.229973629s
FAIL (2025-07-27 00:05:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
g piece...
Jul 27 00:04:09.199649906 (60h28m18.922212946s) [noise=true , address=104.36.82.46:4502 , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200035684 (60h28m18.922598725s) [noise=true , address=79.127.220.101:4048 , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200040494 (60h28m18.922603534s) [noise=true , address=79.127.205.246:4281 , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200375855 (60h28m18.922938895s) [noise=true , address=104.36.82.55:4216 , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.200793785 (60h28m18.923356825s) [noise=true , address=79.127.219.51:4210 , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201093718 (60h28m18.923656758s) [noise=true , address=104.36.82.54:4157 , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201396357 (60h28m18.923959397s) [noise=true , address=79.127.205.233:4646 , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201691182 (60h28m18.924254222s) [noise=true , address=104.36.82.2:4755 , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.201979143 (60h28m18.924542183s) [noise=true , address=79.127.205.235:4666 , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202211828 (60h28m18.924774868s) [noise=true , address=79.127.219.47:4048 , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202447349 (60h28m18.925010389s) [noise=true , address=104.36.82.53:4138 , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202532802 (60h28m18.925095842s) [noise=true , address=79.127.205.245:4245 , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.202798570 (60h28m18.92536161s) [noise=true , address=104.36.82.52:4117 , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203036606 (60h28m18.925599646s) [noise=true , address=79.127.219.38:4010 , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203222462 (60h28m18.925785502s) [noise=true , address=104.36.82.56:4253 , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203476147 (60h28m18.926039177s) [noise=true , address=104.36.82.7:4918 , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:09.203826307 (60h28m18.926389337s) [noise=true , address=156.146.43.225:4448 , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Uploading piece...
Jul 27 00:04:10.303379963 (60h28m20.025942993s) [upload=58000]: Next returned nil segment
Jul 27 00:04:10.303396825 (60h28m20.025959865s) [upload=58000]: Waiting for error group managing segments...
Jul 27 00:04:10.468485635 (60h28m20.191048675s) [noise=true , address=79.127.219.51:4210 , piece_id=SA2Q3RCN4I5BADWQLPIFMAXDT5DGRTNO73V4XYNUD3CPSZNXHGYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000
Jul 27 00:04:10.618950635 (60h28m20.341513666s) [noise=true , address=79.127.205.246:4281 , piece_id=TBFLNGPLPIHQEBWC6QPYBMMGGTR7XMGWRKAQ6QKIUAIONJ4IFGRA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.623707968 (60h28m20.346271008s) [noise=true , address=79.127.205.245:4245 , piece_id=NUR2O6WVYPXSEF3ZYZ3JGV2677DGLGKIIJ4W5AYEGI4U6SPEDHAA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.628858482 (60h28m20.351421522s) [noise=true , address=79.127.205.233:4646 , piece_id=LCSEKEFIVJQ6KEDHCGM36HHCJI7WOKT5ZGBUL6AQY2BP4DTGZMYA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:10.632026083 (60h28m20.354589123s) [noise=true , address=79.127.205.235:4666 , piece_id=JGZE7SDPWWKSSY2VDFDRF275325EYKVS6NKWSEGC6ESTR5RYV7IA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:11.122563874 (60h28m20.845126914s) [noise=true , address=79.127.219.38:4010 , piece_id=IJDYW5SSRM2OR3YFRJLLNIWOZSYYIUAXFUS63SW4244WDB2FJOLQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:11.445377285 (60h28m21.167940325s) [noise=true , address=79.127.220.101:4048 , piece_id=OURI6N4Y56D63UICBUEE7QZF4M3EYYEVJ7AX2O5ZE3ND2EY42V2A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:12.775062857 (60h28m22.497625897s) [noise=true , address=79.127.219.47:4048 , piece_id=CX5MVPF4H5Q36CDDBNSLSIQJBXTN3RNP6LJVLN3NFGTJDZVKFQQQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:12.926457887 (60h28m22.649020927s) [noise=true , address=156.146.43.225:4448 , piece_id=QEF6APTS3KQNEHGSWSL2KLGI7H6MATIYGLCSJ2MC6JT7UKUXB2EQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:04:21.769452468 (60h28m31.492015508s) [noise=true , address=64.49.13.138:4488 , piece_id=5XRHQYWD2BJXGHH7ABFJXDDXIVUS63LRSGRT2SXITYWSJ2WMP2FA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: <nil>
Jul 27 00:05:08.019334096 (60h29m17.741897136s) [noise=true , address=104.36.82.56:4253 , piece_id=MKFS5MKPYG7G4KA3IHLR4S73OZFPOZSFTEQFADPILMSWV7WWM3WA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019386065 (60h29m17.741949105s) [noise=true , address=104.36.82.46:4502 , piece_id=4CRBL5C5EWHPSSGL2CKHW6BSRTOBKGG37TMOA4WOY33GGARTRRNA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019428085 (60h29m17.741991125s) [noise=true , address=104.36.82.53:4138 , piece_id=YGXKUFVOJ75DZQLVDIBL3ZVNIWO2ZILUPQAA7HMNNBQ47RI3EDMA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019461268 (60h29m17.742024309s) [noise=true , address=104.36.82.52:4117 , piece_id=CPIV5XMSHAU2WXMGVYKCLKUIKFYTFH4GRU7XECMKZ2VTQ6COR33A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:104.36.82.52:4117): protocol: rpc: dial tcp 104.36.82.52:4117: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019499592 (60h29m17.742062632s) [noise=true , address=104.36.82.55:4216 , piece_id=6LBUVHE5KEQHNLAHRHWAMZF3WACHJLGKVTPU7I5GQRFNIDWQ734A, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019535661 (60h29m17.742098701s) [noise=true , address=104.36.82.7:4918 , piece_id=UDOCYBP77WWFDMTZQWSRXFAEAO3ZK5ZCTSIVNRYE4PTIIMBQBODA, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:12EMGZ4hpXmBp4TwemLrbKAVtjBmtexMzEahRTzFw42qQipsAG, address:104.36.82.7:4918): protocol: rpc: dial tcp 104.36.82.7:4918: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019575157 (60h29m17.742138197s) [noise=true , address=104.36.82.54:4157 , piece_id=DK2ANOFREWW6INIEKDM3JNIW3WLYYF4QMU5OH5CYEMCGWER2RPZQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019629892 (60h29m17.742192932s) [noise=true , address=104.36.82.2:4755 , piece_id=ICLCWXWWQNCEXEIZPMU4EIZ64X7FBDBUTQE2UAB3246FF7O2Q7JQ, seg_pos={0 0}, upload=58000]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 188.166.225.170:40334->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:08.019646233 (60h29m17.742209273s) [seg_pos={0 0}, upload=58000]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:05:08.246996305 (60h29m17.969559335s) [upload=58000]: Done uploading segments...
Jul 27 00:05:08.247022976 (60h29m17.969586016s) [upload=58000]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.229993076s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [...XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.] duration:
5.944056939s
FAIL (2025-07-27 00:05:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.944068381s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.X] duration:
30.245112591s
FAIL (2025-07-27 00:05:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
UZi3): protocol: expected piece hash; context canceled; EOF
Jul 27 00:03:40.824444038 (60h27m50.547007068s) [seg_pos={0 0}, upload=57993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 27 00:03:40.824514603 (60h27m50.547077643s) [upload=57993]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 27 00:03:40.824519062 (60h27m50.547082092s) [upload=57993]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 27 00:03:41.130112176 (60h27m50.852675216s) [upload=57993]: Done uploading segments...
Jul 27 00:03:41.130131623 (60h27m50.852694663s) [upload=57993]: Upload finished. err: <nil>
Jul 27 00:04:32.085896910 (60h28m41.80845995s) [upload=58008]: Starting upload
Jul 27 00:04:32.085951976 (60h28m41.808515016s) [upload=58008]: Done starting upload
Jul 27 00:04:32.086013413 (60h28m41.808576493s) [upload=58008]: Uploading segments...
Jul 27 00:04:32.086132050 (60h28m41.80869509s) [upload=58008]: Got next segment. Inline: false
Jul 27 00:04:32.086163139 (60h28m41.808726179s) [upload=58008]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:04:32.086168039 (60h28m41.808731069s) [upload=58008]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:04:32.316154546 (60h28m42.038717586s) [seg_pos={0 0}, upload=58008]: Begin upload segment...
Jul 27 00:04:32.316190194 (60h28m42.038753234s) [seg_pos={0 0}, upload=58008]: Done begin upload segment.
Jul 27 00:04:32.316211956 (60h28m42.038774986s) [noise=true , address=104.36.82.10:4079 , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.316550994 (60h28m42.039114034s) [noise=true , address=77.90.31.21:4877 , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.316820280 (60h28m42.03938332s) [noise=true , address=45.91.116.10:4083 , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317062914 (60h28m42.039625954s) [noise=true , address=77.90.31.10:4517 , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317476846 (60h28m42.040039886s) [noise=true , address=45.91.116.6:4974 , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.317739519 (60h28m42.040302559s) [noise=true , address=77.90.31.14:4661 , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318258591 (60h28m42.040821631s) [noise=true , address=77.90.31.16:4733 , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318665460 (60h28m42.0412285s) [noise=true , address=45.91.116.4:4938 , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.318967077 (60h28m42.041530117s) [noise=true , address=77.90.31.16:4734 , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319263524 (60h28m42.041826565s) [noise=true , address=45.91.116.8:4047 , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319569590 (60h28m42.04213263s) [noise=true , address=77.90.31.13:4626 , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.319889312 (60h28m42.042452352s) [noise=true , address=104.36.82.9:4006 , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320214715 (60h28m42.042777755s) [noise=true , address=77.90.31.11:4555 , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320498017 (60h28m42.043061057s) [noise=true , address=77.90.31.15:4699 , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.320812748 (60h28m42.043375788s) [noise=true , address=45.91.116.13:4155 , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:32.321152088 (60h28m42.043715138s) [noise=true , address=104.36.82.15:4257 , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Uploading piece...
Jul 27 00:04:34.057760807 (60h28m43.780323847s) [upload=58008]: Next returned nil segment
Jul 27 00:04:34.057779792 (60h28m43.780342832s) [upload=58008]: Waiting for error group managing segments...
Jul 27 00:04:34.335101806 (60h28m44.057664846s) [noise=true , address=45.91.116.13:4155 , piece_id=O3KD6DANCUFYSTYEDD5ZFUYQZDR3MAUHESK5OXAUZB7EZBPCEAEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.339370523 (60h28m44.061933563s) [noise=true , address=77.90.31.14:4661 , piece_id=A47OUBHKNVLGCIQR6ASCDVXMFVN2YHEYX5OXSF5E5MMS6VQSMKKQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.349741225 (60h28m44.072304255s) [noise=true , address=45.91.116.4:4938 , piece_id=XGGUREMQB7IUS2REQGFMSOFZ7XVTVLXPXILMDD3S2MS6N7L4CXEA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.430079845 (60h28m44.152642875s) [noise=true , address=77.90.31.16:4734 , piece_id=ZGQYC7TZ6S4DHGXRUXUPM2JWW5TXFJPBP7ZCCRQVMFURWZKXJTTA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.475555101 (60h28m44.198118141s) [noise=true , address=77.90.31.21:4877 , piece_id=6RDEYAV6BOBL7S5MOV4PFU7CQ7RY5E5BK2Z5RLNPD5WRAJYMP7PQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.479221716 (60h28m44.201784746s) [noise=true , address=45.91.116.6:4974 , piece_id=7TGQV3NUTHXHPK4D2ATCHS5RYPMO2ZAIBSACLJVMRQHEEOUFEQSA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.497190286 (60h28m44.219753326s) [noise=true , address=77.90.31.11:4555 , piece_id=GV7OJKDR3H45NKLKWJH2FIMFQ6IZ6XBXTPFFKJSOOI2YX7QF7EEQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.499780581 (60h28m44.222343621s) [noise=true , address=45.91.116.10:4083 , piece_id=CMAZSSQVNGW7YLYKDSLWAC7WT4F6CWQU2VWYQHMRPYSP2DTC7NXQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.502073518 (60h28m44.224636558s) [noise=true , address=77.90.31.16:4733 , piece_id=MWTJT7ZR36IJRKQ6EDB5M55HEW4MTXXD7K324XOLOSD4TQSKNQQA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:34.562247958 (60h28m44.284810988s) [noise=true , address=45.91.116.8:4047 , piece_id=43QUJDR6ZR7676AD5M6JNL4SLVK6MF6HRT5GX5D7PF2INO6GEKUQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:04:36.427408290 (60h28m46.14997133s) [noise=true , address=77.90.31.13:4626 , piece_id=H3WZA5CJYXQWWKTKGO7GC7TAHHX43RT4KXFVNDAPN26444VZLFVA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: <nil>
Jul 27 00:05:01.447187046 (60h29m11.169750086s) [noise=true , address=104.36.82.15:4257 , piece_id=C2VF7M7SKHY5ISTRDAXXKITAHHZZG32QDO5XQDXFQ4VHADMXCPBQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:56162->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447238865 (60h29m11.169801905s) [noise=true , address=77.90.31.15:4699 , piece_id=LLXRJR5MUYCIT6UB2A3NEQRVXNGTHLAUODC52QZR43COIRZFPULQ, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447268642 (60h29m11.169831682s) [noise=true , address=104.36.82.10:4079 , piece_id=QCH7JNNFNHZMXBGSYIXW7SUQ2VOKR6IXLYJN5EWP5P6Q33EXYPYA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447297848 (60h29m11.169860898s) [noise=true , address=104.36.82.9:4006 , piece_id=B3ZHSA6BUUCS6KG34JR24HTPUL3EFOD3LK3ZENMMMRCJC7BH3ZXA, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:42962->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447363344 (60h29m11.169926394s) [noise=true , address=77.90.31.10:4517 , piece_id=F62P5KID32MAKPMCBACN4R62AABMSJCKP6Z6HG4673XJFIWOR63Q, seg_pos={0 0}, upload=58008]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58900->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:05:01.447401116 (60h29m11.169964186s) [seg_pos={0 0}, upload=58008]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:05:01.690687079 (60h29m11.413250129s) [upload=58008]: Done uploading segments...
Jul 27 00:05:01.690718899 (60h29m11.413281939s) [upload=58008]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.245134211s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X] duration:
30.001229295s
FAIL (2025-07-27 00:04:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001246707s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.] duration:
7.109247136s
FAIL (2025-07-27 00:04:37 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 7.109270831s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
1.854811833s
FAIL (2025-07-27 00:04:30 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.854880083s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX] duration:
30.227430826s
FAIL (2025-07-27 00:04:19 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.227452569s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.......XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.......] duration:
10.157508953s
FAIL (2025-07-27 00:04:15 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.157530234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.234010391s
FAIL (2025-07-27 00:04:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
429914s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296296305 (60h27m14.018859355s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296448477 (60h27m14.019011567s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296801893 (60h27m14.019364923s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.296916422 (60h27m14.019479462s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297216586 (60h27m14.019779626s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297386952 (60h27m14.019949992s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297594509 (60h27m14.020157549s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297733676 (60h27m14.020296716s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.297961742 (60h27m14.020524782s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298019222 (60h27m14.020582262s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298239854 (60h27m14.020802894s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298338854 (60h27m14.020901894s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298502326 (60h27m14.021065366s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298706026 (60h27m14.021269066s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.298868206 (60h27m14.021431246s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.299167178 (60h27m14.021730229s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:04.299589586 (60h27m14.022152626s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Uploading piece...
Jul 27 00:03:05.385782620 (60h27m15.10834566s) [upload=57985]: Next returned nil segment
Jul 27 00:03:05.385828919 (60h27m15.108391969s) [upload=57985]: Waiting for error group managing segments...
Jul 27 00:03:05.701031808 (60h27m15.423594848s) [noise=true , address=79.127.205.234:4611 , piece_id=BA77RVVXANDBYZ3FSNJIUQAVMEJBMUDBYJTRLDVZIR6OKH5KUP6Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.701960976 (60h27m15.424524006s) [noise=true , address=79.127.205.235:4669 , piece_id=Q5Y7DZ7NSKPD2OG6TJ72DD5UQZ3FEK2DRH3NAPIBW4YHK2FSMXEA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.722094938 (60h27m15.444657968s) [noise=true , address=79.127.205.245:4252 , piece_id=URETIX5E2ELK3ZGA3T3YACMYAFIYUFK3RWJBVBKRRK7LLBV3ZEQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:05.743552713 (60h27m15.466115783s) [noise=true , address=79.127.205.233:4633 , piece_id=3YXOVF2WEGQTCVFQQP2BSMLNIVLIGMNJBCZGB4GAXUE4AYBDPI2A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:06.278512358 (60h27m16.001075398s) [noise=true , address=79.127.219.38:4009 , piece_id=G4WBEZCCNBEM5UCNWGPDB25W22ESZ6HIJ63Y64MEP5QAHMCDVPOA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:08.340018352 (60h27m18.062581392s) [noise=true , address=77.90.31.12:4620 , piece_id=W6RMKIXI2VGQN2F4AUUOGKY3MZ2XL4DFOAZGGZDZMNDL5K2QU4EA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:11.565721846 (60h27m21.288284886s) [noise=true , address=64.49.13.137:4502 , piece_id=IBD3YSDFKQ4RGVIIEXU2KM6GYYUERUKAEKB2DFIUNEFFEP62EYPA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:13.988632217 (60h27m23.711195257s) [noise=true , address=64.49.13.154:4927 , piece_id=OMUQSCOPDV4MZDXANSUPYTISXOQBFSU2DP7RAAHS6BZP7HEXGLGA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:18.683164753 (60h27m28.405727793s) [noise=true , address=45.91.116.14:4206 , piece_id=7T6TU7U3BROVDD34HBRGRY35V7OBK3VH6MKKUC2YNSNBESCNVB7A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:03:30.503118221 (60h27m40.225681261s) [noise=true , address=79.127.201.215:4347 , piece_id=CC3FWPJB5H7VOOTQEKGDH4W37WKG7PVBLGXXAFF7XQMGU3CZUS5Q, seg_pos={0 0}, upload=57985]: Done uploading piece. err: <nil>
Jul 27 00:04:03.190575544 (60h28m12.913138584s) [noise=true , address=104.36.82.54:4171 , piece_id=3NUZJOEGAJSVOKZSEQFHQYWNXRFDN5PFVJXZL4THNQKYDEJ7BZ6A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190656027 (60h28m12.913219067s) [noise=true , address=104.36.82.12:4169 , piece_id=UOZW42AHBA4S2EGZST77THZWOPY5LJEZI2VY2QEFEXIANBORD7WQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: rpc: dial tcp 104.36.82.12:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190692197 (60h28m12.913255237s) [noise=true , address=104.36.82.56:4230 , piece_id=AJKTZJP3OUICAUXFYWV37FNRA44IQIJYTDBDY3SBSVJN6SQP365A, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190745179 (60h28m12.913308219s) [noise=true , address=104.36.82.46:4504 , piece_id=SNAJNZKMMN5Q76CYAMAC6FTXISTZXPF5XOAVO3NOQJDZKSM2YMQQ, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: write tcp 188.166.225.170:37536->104.36.82.46:4504: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190772331 (60h28m12.913335371s) [noise=true , address=104.36.82.40:4962 , piece_id=BYBTCDMRQRQMFEV2PUFMBPU3IEKT6GCTMWZT4NIPUA6ZZXY42SQA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:104.36.82.40:4962): protocol: rpc: dial tcp 104.36.82.40:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190827717 (60h28m12.913390757s) [noise=true , address=77.90.31.10:4543 , piece_id=BTG7VMEAIXE6QUZC2KN6FURFRMLBCFUJS43HNTKSNZ6ZEDIG5YAA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: rpc: dial tcp 77.90.31.10:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190887290 (60h28m12.91345033s) [noise=true , address=104.36.82.47:4552 , piece_id=XWITZMUBQZWZW4PC3K7GEFZYBB2GXLSZ5ESBFRWFGCGQSPS7ILZA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190940382 (60h28m12.913503432s) [noise=true , address=104.36.82.14:4245 , piece_id=JPRJM4XVDB3WECQUDQXZJJN3FH2YFIIUXQTRLIUPQ4J3YK3CFZDA, seg_pos={0 0}, upload=57985]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: rpc: dial tcp 104.36.82.14:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:04:03.190980088 (60h28m12.913543158s) [seg_pos={0 0}, upload=57985]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 27 00:04:03.422740258 (60h28m13.145303289s) [upload=57985]: Done uploading segments...
Jul 27 00:04:03.422770667 (60h28m13.145333717s) [upload=57985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.234036131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X......] duration:
32.74714383s
FAIL (2025-07-27 00:03:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 32.747160172s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......] duration:
6.248021118s
FAIL (2025-07-27 00:03:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.248043411s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.] duration:
5.808010915s
FAIL (2025-07-27 00:03:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload
Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload
Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments...
Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false
Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment...
Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment.
Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment
Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments...
Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments...
Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 5.808034329s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
1.842571085s
FAIL (2025-07-27 00:03:33 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.842596432s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX] duration:
30.237202557s
FAIL (2025-07-27 00:03:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Jul 27 00:03:02.810204684 (60h27m12.532767724s) [upload=57983]: Starting upload
Jul 27 00:03:02.810292011 (60h27m12.532855051s) [upload=57983]: Done starting upload
Jul 27 00:03:02.810399817 (60h27m12.532962857s) [upload=57983]: Uploading segments...
Jul 27 00:03:02.810572368 (60h27m12.533135408s) [upload=57983]: Got next segment. Inline: false
Jul 27 00:03:02.810621742 (60h27m12.533184792s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:03:02.810628255 (60h27m12.533191305s) [upload=57983]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:03:03.043961555 (60h27m12.766524605s) [seg_pos={0 0}, upload=57983]: Begin upload segment...
Jul 27 00:03:03.044018804 (60h27m12.766581854s) [seg_pos={0 0}, upload=57983]: Done begin upload segment.
Jul 27 00:03:03.044047669 (60h27m12.766610709s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044118535 (60h27m12.766681575s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044522038 (60h27m12.767085078s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044574268 (60h27m12.767137298s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044769541 (60h27m12.767332581s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.044941490 (60h27m12.76750454s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045011553 (60h27m12.767574594s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045320225 (60h27m12.767883275s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045335243 (60h27m12.767898293s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045569882 (60h27m12.768132922s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045638323 (60h27m12.768201363s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045839158 (60h27m12.768402198s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.045998822 (60h27m12.768561863s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046054359 (60h27m12.768617409s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046301762 (60h27m12.768864802s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:03.046594272 (60h27m12.769157312s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Uploading piece...
Jul 27 00:03:04.738235055 (60h27m14.460798095s) [upload=57983]: Next returned nil segment
Jul 27 00:03:04.738252338 (60h27m14.460815378s) [upload=57983]: Waiting for error group managing segments...
Jul 27 00:03:05.051026892 (60h27m14.773589932s) [noise=true , address=77.90.31.21:4877 , piece_id=S7P6HTJEKFPVJOTF46YJ56DIB3BEB46D7S57YFA44FDGQEYEPRAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.062839342 (60h27m14.785402382s) [noise=true , address=45.91.116.12:4226 , piece_id=JN6SUAK2Q56SMZ5XYATMKI7WQH2SNMGI5O7WJ5WC6QMDAQWNPWHA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.063702423 (60h27m14.786265463s) [noise=true , address=45.91.116.8:4048 , piece_id=S5UZVWAE47A4SF6STW3JIMXYX3UH7N26JSEXNIC6TBZNO43ND6LQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.065574955 (60h27m14.788137995s) [noise=true , address=77.90.31.14:4663 , piece_id=E6HGUGYPT2ZBJDVJFQXXHH52WMUXREEDYPCCZ4E5ZK6INK6RN76A, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.077566428 (60h27m14.800129479s) [noise=true , address=45.91.116.6:4975 , piece_id=JZCMGNOJ7HR7ZUFYEYX4ZN24V3USMBG5E7DARENGJXLQKTPPPDWQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.084813054 (60h27m14.807376094s) [noise=true , address=45.91.116.7:4010 , piece_id=4O224SZN7G7HDBHLH2KVNVQ7ZCNX6VVYNLNIQHQXLYFGJN6GPBGQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.134443570 (60h27m14.85700661s) [noise=true , address=45.91.116.13:4154 , piece_id=D74ERRC4QINCMSGEEJVDRIQPFES4YBDTRCQRIG4JNSVGYQBRWENQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.163636955 (60h27m14.886199995s) [noise=true , address=45.91.116.10:4083 , piece_id=2IKAYMXMKKK4AHJHDUBIE7JA55LQBLDXJ4LEDLHI37YGALBAR6WQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.219947773 (60h27m14.942510814s) [noise=true , address=45.91.116.6:4974 , piece_id=KPG7ZK5TBMXEC77FF7YUJ3JF6Z3YLNYQLJ7OIHC42NVD46752VPQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:05.307291055 (60h27m15.029854095s) [noise=true , address=45.91.116.4:4937 , piece_id=QQB6HQKYHX62ZS4MQARVE3HTPBA62TSPKVDR425GG4LKNOB7H2PA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: <nil>
Jul 27 00:03:32.172249526 (60h27m41.894812576s) [noise=true , address=77.90.31.17:4771 , piece_id=GT6ZEEW2BPYBW34YKF7M6SRENAICBFXZJAMT262M2UAK6YQE7U6Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172301736 (60h27m41.894864776s) [noise=true , address=77.90.31.16:4733 , piece_id=GARDPK3TQCJLH7PIWMFUWGCEHB5RKGKYNUVYKSWVBSWPEMXETZMQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172341682 (60h27m41.894904722s) [noise=true , address=104.36.82.11:4113 , piece_id=BWLORXDKG4KLJFMUGXMJRL534WVNGMCAJFL7IJJYV6MHF2JEVKOA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172373202 (60h27m41.894936242s) [noise=true , address=77.90.31.10:4520 , piece_id=IEENAQAMQECNSYOWWSRODUE2ESIVN7POCILY55MO4G2O6YEO6QAA, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172408020 (60h27m41.89497106s) [noise=true , address=77.90.31.10:4517 , piece_id=5AJZXW6GW3VGKHNTFCFYXNLBZB77ZJUCFQ2KU53TNAKVZ3SG3O5Q, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172446863 (60h27m41.895009904s) [noise=true , address=104.36.82.15:4257 , piece_id=YMUGSQL4LZGXS5JRY5U6BMV75I7YA6W4LLGTEVSMUOQSI4RK7WUQ, seg_pos={0 0}, upload=57983]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:41230->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:03:32.172462464 (60h27m41.895025504s) [seg_pos={0 0}, upload=57983]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:03:32.405968704 (60h27m42.128531734s) [upload=57983]: Done uploading segments...
Jul 27 00:03:32.405991498 (60h27m42.128554538s) [upload=57983]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.237224419s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.....] duration:
50.911700599s
FAIL (2025-07-27 00:03:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.911720456s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XX] duration:
30.236606689s
FAIL (2025-07-27 00:03:20 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.236625776s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.011069868s
FAIL (2025-07-27 00:03:04 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 2.011098783s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.261749221s
FAIL (2025-07-27 00:02:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
[noise=true , address=104.36.82.52:4090 , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141114954 (60h26m2.863677994s) [noise=true , address=79.127.220.101:4048 , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141201730 (60h26m2.86376477s) [noise=true , address=104.36.82.47:4527 , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141378839 (60h26m2.863941869s) [noise=true , address=79.127.201.215:4356 , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141466387 (60h26m2.864029427s) [noise=true , address=104.36.82.46:4507 , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141624139 (60h26m2.864187179s) [noise=true , address=64.49.13.138:4482 , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141724240 (60h26m2.86428728s) [noise=true , address=79.127.205.245:4223 , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.141888344 (60h26m2.864451384s) [noise=true , address=104.36.82.54:4176 , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142007212 (60h26m2.864570252s) [noise=true , address=79.127.219.47:4048 , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142133743 (60h26m2.864696783s) [noise=true , address=77.90.31.15:4714 , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142227984 (60h26m2.864791024s) [noise=true , address=104.36.82.56:4244 , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142377239 (60h26m2.864940279s) [noise=true , address=104.36.82.7:4931 , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142443786 (60h26m2.865006826s) [noise=true , address=104.36.82.55:4216 , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142668477 (60h26m2.865231517s) [noise=true , address=45.91.116.10:4093 , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142690899 (60h26m2.865253929s) [noise=true , address=79.127.205.233:4639 , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:53.142868359 (60h26m2.865431399s) [noise=true , address=79.127.205.246:4286 , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Uploading piece...
Jul 27 00:01:54.229399462 (60h26m3.951962502s) [upload=57968]: Next returned nil segment
Jul 27 00:01:54.229417798 (60h26m3.951980828s) [upload=57968]: Waiting for error group managing segments...
Jul 27 00:01:54.536628044 (60h26m4.259191084s) [noise=true , address=79.127.205.245:4223 , piece_id=3JPVXCAWUTENDUOM6MKAYTIVLBFKF56LFKX54KFFN2D3XQMMXH2A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.549230276 (60h26m4.271793316s) [noise=true , address=79.127.205.233:4639 , piece_id=WEOARYX5A2LTPHLNVXA5JWIQY6UUIEHITCWJXOSWRGZVLVIEBDMQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.582728527 (60h26m4.305291567s) [noise=true , address=79.127.205.235:4685 , piece_id=3WBPT3C3LCIK2XY25Y5M7FO2CVM4MHZ4DUGBDVSS64EXDI766M3Q, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:54.609006844 (60h26m4.331569884s) [noise=true , address=79.127.205.246:4286 , piece_id=DUOQ2XMLWVTR3MMLZWWGSHHVOCKXH5KCVGFIHRPAX2FSRK3Y4R5A, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:55.637242753 (60h26m5.359805783s) [noise=true , address=64.49.13.138:4482 , piece_id=ODUUY7I4NYEQGIGVAJX54QOEJSWFDFDN57CV5FLITHYNZ3TQLGBA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000
Jul 27 00:01:55.706028969 (60h26m5.428592009s) [noise=true , address=45.91.116.10:4093 , piece_id=BVXUT22ROVXHKH6TMHCDXW4VWTBXH4DRI4KRPRMGXFIAULVHHGOA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:57.495722163 (60h26m7.218285203s) [noise=true , address=79.127.219.47:4048 , piece_id=CL6M44Z4YIN7V6PMXFJV5Z7XQYSHUTARA6WJ7LHGDGBX7YSHDRQA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:58.749704002 (60h26m8.472267042s) [noise=true , address=79.127.220.101:4048 , piece_id=GZKOWKIRMWHDP7ZRZOVJTKX3ZDYNKTZI5NOAGP4GA5HE6NCIJBFQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:01:58.762131119 (60h26m8.484694189s) [noise=true , address=79.127.201.215:4356 , piece_id=IKIJGVU6H3CGZXVMQGBP74KCNXUXDVO4WLFMQY2VJ77PKEXYGJBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: <nil>
Jul 27 00:02:52.042028487 (60h27m1.764591517s) [noise=true , address=104.36.82.46:4507 , piece_id=7QK5PALZ24UJ24ZKCYL7KWZBZQNHUTU3MXUIRT2X2TVZHIH6WOMA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042073464 (60h27m1.764636494s) [noise=true , address=104.36.82.39:4730 , piece_id=FDJH44ZA4Z6LXLNLQFLR5Y2B365AKY5J3X4G3M44SBI5PRMZI5FQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:104.36.82.39:4730): protocol: rpc: dial tcp 104.36.82.39:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042109052 (60h27m1.764672092s) [noise=true , address=104.36.82.52:4090 , piece_id=K5QOFRRUVC4ADFJQOBE3UB3QQ42ESY3665VYOKAFI6B6GAIBGPBQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 188.166.225.170:32790->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042141123 (60h27m1.764704163s) [noise=true , address=104.36.82.54:4176 , piece_id=MRKFR3ISOFG4NYEN2QWJHKY6BWNNW2D5OGULVDKIANL4EXZNZ4CQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042285480 (60h27m1.76484852s) [noise=true , address=104.36.82.7:4931 , piece_id=OF5JE2NSNOL544CTHTMJSVYWNWK4PPHT4ZK2PPBCYAQZWMIH67JQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:125rm4HueKsNdwL3eCTCSWoPseR2LQWcsE3jNPpxSfYQQorj9LL, address:104.36.82.7:4931): protocol: rpc: dial tcp 104.36.82.7:4931: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042300318 (60h27m1.764863358s) [noise=true , address=104.36.82.47:4527 , piece_id=UB2KCS7NPHD4R23AVXWHUINSTHT6J55ZYGDOZ26YYNSCU46AX5JA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042359180 (60h27m1.76492222s) [noise=true , address=104.36.82.55:4216 , piece_id=JC363CSP7V63JEKEJTWNUFNJRDXYY6P3EPOIZMUI5EHKBXTVBXVQ, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042365232 (60h27m1.764928272s) [noise=true , address=77.90.31.15:4714 , piece_id=C2OQODDYKPTL272J5DVSWFV56KTCKSVFGXTET5GIEXLRGMEZIJPA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: read tcp 188.166.225.170:58426->77.90.31.15:4714: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042394899 (60h27m1.764957939s) [noise=true , address=104.36.82.56:4244 , piece_id=O4ZV6URO6CEXW5NEMH4L2BN3Y7JF2SUTRXMFXCJUOHLZA7PX6ETA, seg_pos={0 0}, upload=57968]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 188.166.225.170:53112->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:52.042423023 (60h27m1.764986063s) [seg_pos={0 0}, upload=57968]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Jul 27 00:02:52.301418572 (60h27m2.023981632s) [upload=57968]: Done uploading segments...
Jul 27 00:02:52.301447227 (60h27m2.024010267s) [upload=57968]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 1m0.261765342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.X] duration:
30.234335876s
FAIL (2025-07-27 00:02:50 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.234357208s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXX] duration:
30.234483068s
FAIL (2025-07-27 00:02:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
26260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:02:09.926623655 (60h26m19.649186695s) [upload=57970]: Starting upload
Jul 27 00:02:09.926674673 (60h26m19.649237863s) [upload=57970]: Done starting upload
Jul 27 00:02:09.926868904 (60h26m19.649431934s) [upload=57970]: Uploading segments...
Jul 27 00:02:09.926902659 (60h26m19.649465689s) [upload=57970]: Got next segment. Inline: false
Jul 27 00:02:09.926938417 (60h26m19.649501457s) [upload=57970]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:02:09.926942434 (60h26m19.649505474s) [upload=57970]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:02:10.159693471 (60h26m19.882256511s) [seg_pos={0 0}, upload=57970]: Begin upload segment...
Jul 27 00:02:10.159735501 (60h26m19.882298541s) [seg_pos={0 0}, upload=57970]: Done begin upload segment.
Jul 27 00:02:10.159757503 (60h26m19.882320543s) [noise=true , address=45.91.116.7:4010 , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.159819051 (60h26m19.882382091s) [noise=true , address=77.90.31.21:4877 , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160178649 (60h26m19.882741679s) [noise=true , address=77.90.31.12:4590 , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160383892 (60h26m19.882946922s) [noise=true , address=77.90.31.15:4699 , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160457392 (60h26m19.883020432s) [noise=true , address=104.36.82.12:4149 , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160718953 (60h26m19.883281993s) [noise=true , address=45.91.116.14:4189 , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.160747237 (60h26m19.883310277s) [noise=true , address=77.90.31.13:4627 , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161021091 (60h26m19.883584131s) [noise=true , address=104.36.82.14:4221 , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161057069 (60h26m19.883620109s) [noise=true , address=104.36.82.3:4765 , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161365109 (60h26m19.883928139s) [noise=true , address=45.91.116.13:4154 , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161371051 (60h26m19.883934091s) [noise=true , address=77.90.31.18:4805 , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161619686 (60h26m19.884182887s) [noise=true , address=104.36.82.16:4293 , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161651557 (60h26m19.884214597s) [noise=true , address=104.36.82.8:4969 , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161875946 (60h26m19.884438976s) [noise=true , address=104.36.82.7:4909 , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.161921764 (60h26m19.884484794s) [noise=true , address=45.91.116.8:4048 , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:10.162120615 (60h26m19.884683645s) [noise=true , address=45.91.116.10:4083 , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Uploading piece...
Jul 27 00:02:11.908387111 (60h26m21.630950151s) [upload=57970]: Next returned nil segment
Jul 27 00:02:11.908404243 (60h26m21.630967283s) [upload=57970]: Waiting for error group managing segments...
Jul 27 00:02:12.184893462 (60h26m21.907456502s) [noise=true , address=45.91.116.7:4010 , piece_id=XWF3L46YDSC7AIOEVNGPQOVMERHHMIKNYSG4WKU5PUDNS5M4G5RA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.185162627 (60h26m21.907725657s) [noise=true , address=77.90.31.18:4805 , piece_id=R7MNKKGD4KNZN2O2GISBYWC5UAT6E4BZSEMLBJ5A2GX6RMWT7Z3A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.230461587 (60h26m21.953024617s) [noise=true , address=45.91.116.10:4083 , piece_id=FEFTOSG4IUOCOZBO6GE3CS2CJFWG5YBSZYRKUWAX2HL6AFNCIXKQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.256849564 (60h26m21.979412604s) [noise=true , address=45.91.116.13:4154 , piece_id=YJ4LZHAH37F5BANUV6GCACOISZTASNEMACCOJYD7TEF6OKXWIE3Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.262586270 (60h26m21.98514931s) [noise=true , address=77.90.31.13:4627 , piece_id=UC7YLH2KTADMCU6I6IVBCYEE6YCHZ5QPDZVY6KEQQJIOO6TXXBSQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.283764439 (60h26m22.006327479s) [noise=true , address=77.90.31.21:4877 , piece_id=652NXHGB7IWKLLETXM3AWOAJKG4BVHVG4RWCUYN7QFITM6FFNO6A, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.353431912 (60h26m22.075994953s) [noise=true , address=45.91.116.8:4048 , piece_id=DRQNEJ4VDZ2NZJSLM74X7CSEIWV42UHY43UHRXWSFP7Q27HVIZCQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:12.482312036 (60h26m22.204875076s) [noise=true , address=45.91.116.14:4189 , piece_id=RJTLGDCDYZCB2O5RT7P26VYGLLYPTGUKVVR4BTAVOEESG2OPPMEQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:13.616478367 (60h26m23.339041407s) [noise=true , address=77.90.31.12:4590 , piece_id=6UWSO373PD7QOLHFXDGFK4IGXIZIIK7NKCDWTKU5QSKMBYWDCLTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: <nil>
Jul 27 00:02:39.279759003 (60h26m49.002322084s) [noise=true , address=104.36.82.16:4293 , piece_id=5RCVOAJXWD4AW27GC636RYEOD6RD4CK3UYBY5H5LEBBNYXRXA7KA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279833485 (60h26m49.002396525s) [noise=true , address=104.36.82.3:4765 , piece_id=MUPL4H5IBYMBKGZ6JAQR42PW5ECT46XFN73D2AA72TL3WEKHTHZQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:38402->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279870246 (60h26m49.002433286s) [noise=true , address=104.36.82.14:4221 , piece_id=OJCZY4CFUWOU2A3VRBDG4VXAUAFNY6MGIJAK6YBIDHW4CJ7UWZTA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:44646->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279916685 (60h26m49.002479725s) [noise=true , address=104.36.82.7:4909 , piece_id=DZEASKNQIZ2R74GRZSROTWEG4FL6ZDVXJHUANZU3ASV6ORPJADNQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279955159 (60h26m49.002518199s) [noise=true , address=77.90.31.15:4699 , piece_id=O5TWOAILIPQ7H4EIXJXRSAFN2OGOJPXJYYUMFHQTQJ4ATBKQWGCA, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.279987791 (60h26m49.002550821s) [noise=true , address=104.36.82.8:4969 , piece_id=5ZDNEJMOBATMM37IKWEYC7KPWRI3N66CEFBUUSRUWVHJXOSNEQ5Q, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41792->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280021024 (60h26m49.002584074s) [noise=true , address=104.36.82.12:4149 , piece_id=36K5FPFGBH4CZBFTNK6MEMTKYZKD3DD7YOVIOCA4FEGECO5WD7RQ, seg_pos={0 0}, upload=57970]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:33538->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:39.280061913 (60h26m49.002624983s) [seg_pos={0 0}, upload=57970]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Jul 27 00:02:39.511968572 (60h26m49.234531612s) [upload=57970]: Done uploading segments...
Jul 27 00:02:39.511995574 (60h26m49.234558614s) [upload=57970]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.234525259s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X.....] duration:
6.329550432s
FAIL (2025-07-27 00:02:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.329570659s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X....] duration:
32.728698643s
FAIL (2025-07-27 00:02:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 32.728720753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.] duration:
7.023299469s
FAIL (2025-07-27 00:02:09 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 7.023323927s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXX] duration:
30.232953879s
FAIL (2025-07-27 00:02:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
n, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments...
Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:01:35.539401218 (60h25m45.261964258s) [upload=57962]: Starting upload
Jul 27 00:01:35.539463527 (60h25m45.262026557s) [upload=57962]: Done starting upload
Jul 27 00:01:35.539640455 (60h25m45.262203496s) [upload=57962]: Uploading segments...
Jul 27 00:01:35.539671956 (60h25m45.262234996s) [upload=57962]: Got next segment. Inline: false
Jul 27 00:01:35.539692415 (60h25m45.262255445s) [upload=57962]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:01:35.539696553 (60h25m45.262259593s) [upload=57962]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:01:35.773032700 (60h25m45.49559574s) [seg_pos={0 0}, upload=57962]: Begin upload segment...
Jul 27 00:01:35.773073748 (60h25m45.495636778s) [seg_pos={0 0}, upload=57962]: Done begin upload segment.
Jul 27 00:01:35.773095660 (60h25m45.49565869s) [noise=true , address=77.90.31.21:4878 , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.773470618 (60h25m45.496033658s) [noise=true , address=45.91.116.7:4011 , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.773974211 (60h25m45.496537251s) [noise=true , address=77.90.31.19:4842 , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774008596 (60h25m45.496571676s) [noise=true , address=104.36.82.11:4114 , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774410496 (60h25m45.496973536s) [noise=true , address=77.90.31.16:4733 , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774513844 (60h25m45.497076884s) [noise=true , address=45.91.116.13:4153 , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774727582 (60h25m45.497290612s) [noise=true , address=45.91.116.13:4154 , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.774883190 (60h25m45.49744623s) [noise=true , address=45.91.116.4:4937 , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775210867 (60h25m45.497773907s) [noise=true , address=45.91.116.14:4191 , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775268106 (60h25m45.497831146s) [noise=true , address=104.36.82.12:4150 , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775591756 (60h25m45.498154796s) [noise=true , address=45.91.116.6:4974 , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.775923420 (60h25m45.49848646s) [noise=true , address=45.91.116.9:4263 , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776273600 (60h25m45.49883664s) [noise=true , address=77.90.31.15:4699 , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776579666 (60h25m45.499142706s) [noise=true , address=77.90.31.10:4518 , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.776864541 (60h25m45.499427581s) [noise=true , address=77.90.31.12:4592 , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:35.777213028 (60h25m45.499776068s) [noise=true , address=104.36.82.3:4765 , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Uploading piece...
Jul 27 00:01:37.516632604 (60h25m47.239195644s) [upload=57962]: Next returned nil segment
Jul 27 00:01:37.516644707 (60h25m47.239207747s) [upload=57962]: Waiting for error group managing segments...
Jul 27 00:01:37.793047892 (60h25m47.515610932s) [noise=true , address=45.91.116.13:4154 , piece_id=EHCSVULC34I3IVOXJI2MNC2WKICRAYPAKWBN6HH5NBJONRZWOUWA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.877611314 (60h25m47.600174354s) [noise=true , address=45.91.116.14:4191 , piece_id=CVN3GV3R3SCNXADHVFIRY2I4CUFLVGAILDW5Z4UVM6RYO67KEAVQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.886850622 (60h25m47.609413662s) [noise=true , address=45.91.116.7:4011 , piece_id=TRMBI7ZGV44UZSFH2OK5ZQLHU73Y7EBOEGSVPS66PE362SYGR2NQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.887828463 (60h25m47.610391503s) [noise=true , address=45.91.116.6:4974 , piece_id=COXYBI3EV44XEVXMBOE2KQLVO4VZ3IJZ4EZV7YQ5FHULONLG4C5A, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.894226173 (60h25m47.616789213s) [noise=true , address=77.90.31.21:4878 , piece_id=6QMSHLGATFJI2RJB2RXRXVOX32DLYJ4CAUWKMECRNVN3BVLWD5TQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.954015056 (60h25m47.676578096s) [noise=true , address=45.91.116.4:4937 , piece_id=ZKCR7OJATCJCLB5IKRIH6STREWE2HYDY5HMV5WLVGV2BGR2CKKPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.954674427 (60h25m47.677237468s) [noise=true , address=45.91.116.13:4153 , piece_id=BHKBBIMCBTGBD4LLUQ66XLU5G56W6WMBDEA6BSBXHT43KAJ3AABA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:37.968328110 (60h25m47.690891161s) [noise=true , address=45.91.116.9:4263 , piece_id=BJZFJ46WWQHMFRXHSPDQHAX2SXM7THZCLWUZXQSMTT6NBFO2PLPQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.090232961 (60h25m48.812796001s) [noise=true , address=77.90.31.12:4592 , piece_id=QVXA7AW33VV5WQOLNCCALOSTCLSTO42UJXW3I5SN5JACWDT5ALNQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.245905354 (60h25m48.968468394s) [noise=true , address=77.90.31.15:4699 , piece_id=ZB632RZCCTE5WTWNS4V7KW7K7LBOKYRFRKEVNPES53JGDXYJFKLQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:01:39.909345256 (60h25m49.631908297s) [noise=true , address=77.90.31.19:4842 , piece_id=3K3C3464JJP6ZYSMUZQQEAAZN63QRY5IWN6KXJXEVEROIOSTCCRA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: <nil>
Jul 27 00:02:04.895836786 (60h26m14.618399826s) [noise=true , address=104.36.82.3:4765 , piece_id=LTVMA2Z4YOKY7XPXOYOF3HRA4RCVJXSXXZDSVOHYPGOC6MSPOQCQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:57196->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.895965743 (60h26m14.618528783s) [noise=true , address=77.90.31.10:4518 , piece_id=N7766F6J3N523NKJ555UIAERFJCGEJE6IDKIM2D4VAFNVGI7CDUA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.895998005 (60h26m14.618561045s) [noise=true , address=104.36.82.11:4114 , piece_id=EZQRN76E4PG6DTIWXM74WECWNEMIJOGL6PM6HZXQNOPFWSK6A3IA, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:37496->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896051637 (60h26m14.618614678s) [noise=true , address=77.90.31.16:4733 , piece_id=KLVNYWJ5YO3IE74Q2LGIVR77HWFJ55EAUA362HJXJKRYS7W73YZQ, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:45436->77.90.31.16:4733: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896131230 (60h26m14.61869428s) [noise=true , address=104.36.82.12:4150 , piece_id=KG27XCZ3YEO27JFXP3XDWZHKSR3XGDH7OQWL57IXZG6STDAVTU4Q, seg_pos={0 0}, upload=57962]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:02:04.896156248 (60h26m14.618719288s) [seg_pos={0 0}, upload=57962]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:02:05.126235353 (60h26m14.848798393s) [upload=57962]: Done uploading segments...
Jul 27 00:02:05.126260180 (60h26m14.84882322s) [upload=57962]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.232968676s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X] duration:
30.235725345s
FAIL (2025-07-27 00:01:56 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.235739561s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.232792664s
FAIL (2025-07-27 00:01:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
, upload=57952]: Uploading piece...
Jul 27 00:00:52.690310647 (60h25m2.412873687s) [noise=true , address=45.91.116.14:4194 , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690629588 (60h25m2.413192628s) [noise=true , address=79.127.219.47:4048 , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690755619 (60h25m2.413318659s) [noise=true , address=104.36.82.47:4526 , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.690902470 (60h25m2.41346551s) [noise=true , address=79.127.219.38:4018 , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691053178 (60h25m2.413616218s) [noise=true , address=104.36.82.52:4095 , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691357762 (60h25m2.413920802s) [noise=true , address=45.91.116.14:4206 , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691444227 (60h25m2.414007267s) [noise=true , address=104.36.82.53:4149 , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691650000 (60h25m2.41421304s) [noise=true , address=79.127.205.234:4608 , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691722409 (60h25m2.414285449s) [noise=true , address=79.127.220.97:4361 , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.691963541 (60h25m2.414526571s) [noise=true , address=77.90.31.10:4540 , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692069282 (60h25m2.414632322s) [noise=true , address=79.127.201.215:4352 , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692265959 (60h25m2.414828999s) [noise=true , address=64.49.13.138:4482 , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692443869 (60h25m2.415006899s) [noise=true , address=156.146.43.225:4448 , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692516518 (60h25m2.415079559s) [noise=true , address=64.49.13.138:4483 , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692758370 (60h25m2.41532141s) [noise=true , address=104.36.82.12:4168 , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692778279 (60h25m2.415341309s) [noise=true , address=104.36.82.46:4499 , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:52.692999933 (60h25m2.415562973s) [noise=true , address=104.36.82.49:4998 , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Uploading piece...
Jul 27 00:00:53.707783843 (60h25m3.430346883s) [noise=true , address=79.127.219.47:4048 , piece_id=XEZRMXHY2BCSF2U4MZRIUMBL2CP5HCWDWDSDJDE4APCKVQKCPXHA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000
Jul 27 00:00:53.808335285 (60h25m3.530898325s) [upload=57952]: Next returned nil segment
Jul 27 00:00:53.808353410 (60h25m3.53091645s) [upload=57952]: Waiting for error group managing segments...
Jul 27 00:00:53.894991431 (60h25m3.617554461s) [noise=true , address=79.127.220.97:4361 , piece_id=X2S7RYPKUY4Y3N52657RNKIGEOP5QH2Z7IQ2NFFLUFYJLTVR5JBA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000
Jul 27 00:00:54.294741272 (60h25m4.017304302s) [noise=true , address=79.127.205.234:4608 , piece_id=3IWACSSRKOK4TNXD6M6HPXMIDLZ3D7RYT7ZMEN2NRFYPSCPAO54A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.372553368 (60h25m4.095116398s) [noise=true , address=64.49.13.138:4482 , piece_id=IIKATN7PR3P3J7WIWNPXQBUVU3UC7UQ452FR66ECU67DHUMHCAUQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.551991192 (60h25m4.274554232s) [noise=true , address=79.127.219.38:4018 , piece_id=DQCD6MATCWGBTOKU2ORXI5T5UKBLO672JZ2L2CZU7USVD37ELN6A, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:54.786651822 (60h25m4.509214872s) [noise=true , address=45.91.116.14:4194 , piece_id=TYJGKSQW4S4HIXPHGPRARII7GZVNAJKP6VNFPUFWSLVKX3TYB5OQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:56.389526811 (60h25m6.112089851s) [noise=true , address=45.91.116.14:4206 , piece_id=64MVCQODQT7GZYIUY5TYGWH7QBIJWMQVXYZIW3W32XCREX4NCEOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:57.469684549 (60h25m7.192247589s) [noise=true , address=156.146.43.225:4448 , piece_id=ZNG3NBZUW3HYXJVSRVGQMXY274HYQVG432KJ4PYFIH57V3WKVUSQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:57.904513196 (60h25m7.627076236s) [noise=true , address=79.127.201.214:4321 , piece_id=FQBU6OOVFDLBJCM44XHHWTGH3BOHMTC2XOBJH3L6DNOQAAFIQHOQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:00:59.255459436 (60h25m8.978022476s) [noise=true , address=64.49.13.138:4483 , piece_id=FV5TAIQOVWHZO2O2JUIF3FJBLRJUJSCG7AJMVJ2BDAIZNEF5ZJYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:01:20.907151736 (60h25m30.629714776s) [noise=true , address=79.127.201.215:4352 , piece_id=GKUMD6VFDCS437W74JGMZCRAURDAHMOJ3ZVY37MYFT5QSZLHYH6Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: <nil>
Jul 27 00:01:51.585590532 (60h26m1.308153573s) [noise=true , address=104.36.82.53:4149 , piece_id=MHL4WYOTTQWKHDNBZQNVJKYDT47BINZEZMEJ6K3X75L7ETIFZDYQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585652411 (60h26m1.308215451s) [noise=true , address=77.90.31.10:4540 , piece_id=PPNNPMJO5XVKD5F6ACOCOQAVSTLFOUUNUDIQR3BUULQQG3ZL4EQQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 188.166.225.170:37456->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585885096 (60h26m1.308448136s) [noise=true , address=104.36.82.46:4499 , piece_id=VU3DEG57D55CO3B2DQUYYZGCSE3B7LFG27TQ7XXPZUC453PIFC7Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: write tcp 188.166.225.170:35230->104.36.82.46:4499: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585951222 (60h26m1.308514263s) [noise=true , address=104.36.82.52:4095 , piece_id=U2K75F3UGFAMK7IRQCUS4LN2XLI5ALYFLDLN4CSX3ICO2ULZM62Q, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: rpc: dial tcp 104.36.82.52:4095: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.585973626 (60h26m1.308536686s) [noise=true , address=104.36.82.12:4168 , piece_id=VLPITRMU5YBZ3IH3ZZRKYH6AYLETBDXJ2WYRSUBRH7ZXR2BQWGHQ, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586015997 (60h26m1.308579037s) [noise=true , address=104.36.82.49:4998 , piece_id=YI6E4MBIDF6GSMJJ4AI7LVTAPBW2CZMVGOZCGQUQDL4DDS3TFBLA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:104.36.82.49:4998): protocol: rpc: dial tcp 104.36.82.49:4998: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586060892 (60h26m1.308623932s) [noise=true , address=104.36.82.47:4526 , piece_id=5AL7R5RIA557DPRKF5BXHM472PYTXRF7OVU2MVLX5YSUANMUADWA, seg_pos={0 0}, upload=57952]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:33012->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:51.586097172 (60h26m1.308660243s) [seg_pos={0 0}, upload=57952]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:01:51.816141159 (60h26m1.538704189s) [upload=57952]: Done uploading segments...
Jul 27 00:01:51.816172810 (60h26m1.53873585s) [upload=57952]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.232819476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
1.862142079s
FAIL (2025-07-27 00:01:51 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.862169621s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X....] duration:
7.628504267s
FAIL (2025-07-27 00:01:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.628519625s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...........XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...] duration:
22.886072806s
FAIL (2025-07-27 00:01:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.886084397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX] duration:
30.23914945s
FAIL (2025-07-27 00:01:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
ent: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments...
Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Jul 27 00:01:05.077717119 (60h25m14.800280159s) [upload=57953]: Starting upload
Jul 27 00:01:05.077788786 (60h25m14.800351827s) [upload=57953]: Done starting upload
Jul 27 00:01:05.077967438 (60h25m14.800530478s) [upload=57953]: Uploading segments...
Jul 27 00:01:05.078083480 (60h25m14.80064652s) [upload=57953]: Got next segment. Inline: false
Jul 27 00:01:05.078125841 (60h25m14.800688881s) [upload=57953]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:01:05.078131832 (60h25m14.800694872s) [upload=57953]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:01:05.310276219 (60h25m15.03283926s) [seg_pos={0 0}, upload=57953]: Begin upload segment...
Jul 27 00:01:05.310332166 (60h25m15.032895207s) [seg_pos={0 0}, upload=57953]: Done begin upload segment.
Jul 27 00:01:05.310344830 (60h25m15.03290787s) [noise=true , address=77.90.31.12:4590 , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310361623 (60h25m15.032924663s) [noise=true , address=104.36.82.10:4078 , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310472134 (60h25m15.033035174s) [noise=true , address=45.91.116.11:4118 , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.310905342 (60h25m15.033468382s) [noise=true , address=104.36.82.16:4293 , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311221878 (60h25m15.033784918s) [noise=true , address=104.36.82.8:4969 , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311550277 (60h25m15.034113327s) [noise=true , address=45.91.116.9:4261 , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.311893673 (60h25m15.034456713s) [noise=true , address=45.91.116.10:4081 , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312251939 (60h25m15.034814989s) [noise=true , address=77.90.31.15:4699 , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312583363 (60h25m15.035146403s) [noise=true , address=45.91.116.6:4975 , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.312902724 (60h25m15.035465774s) [noise=true , address=104.36.82.2:4729 , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.313247313 (60h25m15.035810353s) [noise=true , address=45.91.116.8:4045 , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.313579468 (60h25m15.036142508s) [noise=true , address=45.91.116.13:4154 , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.314389629 (60h25m15.036952679s) [noise=true , address=104.36.82.7:4910 , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.315145515 (60h25m15.037708565s) [noise=true , address=77.90.31.19:4841 , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.316289474 (60h25m15.038852524s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:05.316765935 (60h25m15.039328985s) [noise=true , address=45.91.116.12:4227 , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Uploading piece...
Jul 27 00:01:07.052152676 (60h25m16.774715716s) [upload=57953]: Next returned nil segment
Jul 27 00:01:07.052172283 (60h25m16.774735323s) [upload=57953]: Waiting for error group managing segments...
Jul 27 00:01:07.346109275 (60h25m17.068672315s) [noise=true , address=45.91.116.9:4261 , piece_id=UYO2DHI47LBHT7M2I5SJI6D5AMNIT3D5YAF7PKRER32CXWIA3SVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.346622458 (60h25m17.069185498s) [noise=true , address=45.91.116.8:4045 , piece_id=X76UK7TA3FALINWATVD2PUOCH6C7GOXUUA6A4PRRLYMUP4NWE6EQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.420735295 (60h25m17.143298325s) [noise=true , address=45.91.116.10:4081 , piece_id=U2IWYB4YI5NJNPDCXNWL6VLPTKG4KB22N4AS2OCWQLNMQF5EXJGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.439976829 (60h25m17.162539869s) [noise=true , address=45.91.116.12:4227 , piece_id=LD3IFVXN5C2JTFPTDRWCLC4OOHEK7FCM4ILDGZPHVBSVHJV6PUGQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.445103078 (60h25m17.167666118s) [noise=true , address=45.91.116.13:4154 , piece_id=TXHYPRGEHV6YTKDDFSHJUBLL5DMSPRRY47E4V3GNFSDMM3XAAEYQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.455448041 (60h25m17.178011081s) [noise=true , address=77.90.31.12:4590 , piece_id=O5BFSXRSXV4JKYTLMLQAKOBSSB4SDOPVGO33NOIZM3UY7RZASF7A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.480860661 (60h25m17.203423732s) [noise=true , address=45.91.116.11:4118 , piece_id=G65HGPM4NA5VAA6ZZC4ZPBA7WZXYUJFL4L6SRBERNWKDDFC2K2VQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.494123216 (60h25m17.216686246s) [noise=true , address=45.91.116.6:4975 , piece_id=QVNRLZPHJHE4HC2LMUZI7SUUYDRU4QUWIEDSSOHT7WEO2YCKKQKA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:07.505061495 (60h25m17.227624535s) [noise=true , address=77.90.31.18:4806 , piece_id=DLXGBEA7TAOR7ZERDYLPYL5KC752SOCWDIQJXARXZX7HS4WJFVVA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:08.692327849 (60h25m18.414890889s) [noise=true , address=77.90.31.19:4841 , piece_id=AZ3I45WCDJOHYSWEFDP7RHSN6HVBYJXBG4YBZAMK4A2W2JWXGOZQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:08.870774407 (60h25m18.593337457s) [noise=true , address=77.90.31.15:4699 , piece_id=Y44IUYJOMOZQXQANM76NN4UDF7UM23TF3WE7R6ECOYT57HUNWBJA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: <nil>
Jul 27 00:01:34.428248528 (60h25m44.150811578s) [noise=true , address=104.36.82.10:4078 , piece_id=A2T2MYRP3LDEU6FPS4S54ES3DB3QDPCM2UR6US4WRL35REXIXJWA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:38342->104.36.82.10:4078: use of closed network connection; conn closed: op
Jul 27 00:01:34.428315156 (60h25m44.150878226s) [noise=true , address=104.36.82.2:4729 , piece_id=QWNKXAWESODBBMLLAE2JPPA32AE7ZHSE33CYMA5WRPNVEO6FMOFA, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:49100->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:34.428374339 (60h25m44.150937379s) [noise=true , address=104.36.82.8:4969 , piece_id=7TK5AMFAZXT7YYQUYMEXIKPTSJ3R3JVDBK4SPZQLCKU3C32SXELQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428446246 (60h25m44.151009296s) [noise=true , address=104.36.82.16:4293 , piece_id=5GUVVMFYPWHG6RHJLOI7C52CF4DH5FD7KKEUNC6FMMEMEHQY4VCQ, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:34.428586766 (60h25m44.151149836s) [noise=true , address=104.36.82.7:4910 , piece_id=VSIEB5AYT5OF2BBWNYBA3RUXO4KLVGEXFKTUKGUXVGDPTVZENJ3A, seg_pos={0 0}, upload=57953]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:34102->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:01:34.428622484 (60h25m44.151185544s) [seg_pos={0 0}, upload=57953]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:34.665341051 (60h25m44.387904092s) [upload=57953]: Done uploading segments...
Jul 27 00:01:34.665367592 (60h25m44.387930632s) [upload=57953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.239168005s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
1.949046689s
FAIL (2025-07-27 00:01:21 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.949078792s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..] duration:
29.413031147s
FAIL (2025-07-27 00:01:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.413046005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXX] duration:
30.235163086s
FAIL (2025-07-27 00:01:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments...
Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:00:34.625856171 (60h24m44.348419211s) [upload=57946]: Starting upload
Jul 27 00:00:34.625958327 (60h24m44.348521407s) [upload=57946]: Done starting upload
Jul 27 00:00:34.626040784 (60h24m44.348603824s) [upload=57946]: Uploading segments...
Jul 27 00:00:34.626264823 (60h24m44.348827863s) [upload=57946]: Got next segment. Inline: false
Jul 27 00:00:34.626311623 (60h24m44.348874663s) [upload=57946]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:00:34.626317774 (60h24m44.348880844s) [upload=57946]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:00:34.865594065 (60h24m44.588157106s) [seg_pos={0 0}, upload=57946]: Begin upload segment...
Jul 27 00:00:34.865647087 (60h24m44.588210127s) [seg_pos={0 0}, upload=57946]: Done begin upload segment.
Jul 27 00:00:34.865670111 (60h24m44.588233141s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.865708626 (60h24m44.588271666s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866075106 (60h24m44.588638146s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866256885 (60h24m44.588819925s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866404937 (60h24m44.588967978s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866648443 (60h24m44.589211484s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866667950 (60h24m44.58923099s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.866930773 (60h24m44.589493813s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867221260 (60h24m44.58978429s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867547112 (60h24m44.590110152s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.867777634 (60h24m44.590340674s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868010669 (60h24m44.590573699s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868350270 (60h24m44.59091331s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868577584 (60h24m44.591140624s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.868844886 (60h24m44.591407926s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:34.869181019 (60h24m44.591744049s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Uploading piece...
Jul 27 00:00:36.571647152 (60h24m46.294210192s) [upload=57946]: Next returned nil segment
Jul 27 00:00:36.571683281 (60h24m46.294246331s) [upload=57946]: Waiting for error group managing segments...
Jul 27 00:00:36.869992438 (60h24m46.592555478s) [noise=true , address=45.91.116.11:4118 , piece_id=3VH3DMM5GJW5KDQOIXDZEKPCNDNRK7EFSFIXWKJUGJ2ESTJX4WTA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.887108205 (60h24m46.609671245s) [noise=true , address=45.91.116.14:4190 , piece_id=RT6IBMEH3ZYJWL4N4GVPCMDC7DOPJ253JHMXTK2ENCUXUZDH45JQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.907036524 (60h24m46.629599574s) [noise=true , address=45.91.116.8:4045 , piece_id=FM2KVT4LFCMRS7UYNFHWLX2RN6TBFP2XKE6ORDXBVZIQVQGXV53A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.958513032 (60h24m46.681076072s) [noise=true , address=45.91.116.7:4009 , piece_id=A2UZZAE6RPWYWPCRC4T77AUPJGORFRM6PGE37RHVV44H3YBXP3VQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.973207827 (60h24m46.695770867s) [noise=true , address=77.90.31.13:4627 , piece_id=6E53RZVTWQRYSRBJSYXLZNSJ4DG5XXSMF3AZMJUI3JW4ZFVQWM5Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.974133909 (60h24m46.696696949s) [noise=true , address=45.91.116.13:4154 , piece_id=5X3MZBZQ23GHERSZSSILI7ZPAKGMGZGBLGT6IPBYGNHJOZDF4VRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.977135692 (60h24m46.699698732s) [noise=true , address=45.91.116.13:4153 , piece_id=2JZCYLREGHZDUC3AILNFPSV7LODH24XIBWURZOOTRF7ZWRSGBQRA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:36.978503389 (60h24m46.701066429s) [noise=true , address=77.90.31.18:4805 , piece_id=5SHMTSK2MNV5DXPMHOLLVGB662H62TRGBYABEIKN6HDZOT42PXYA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.016312929 (60h24m46.738875969s) [noise=true , address=45.91.116.5:4297 , piece_id=X3QOPB5WBXPTQW3TAOIYHGXFN335Q4VCLL6QGXFI76K34APZSIXA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.021475726 (60h24m46.744038766s) [noise=true , address=45.91.116.12:4226 , piece_id=WMADAA2TBCIUFUIV3GRWO2VUB6BEEWYBSN72FMKNCYWP4C5SKP2A, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:00:37.036167487 (60h24m46.758730527s) [noise=true , address=45.91.116.6:4973 , piece_id=BG5DUZJAB2GZVIMSGWLWM5UUUKXVBIDDSOD74MK6WSR2NUWL44NA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: <nil>
Jul 27 00:01:03.968881072 (60h25m13.691444112s) [noise=true , address=104.36.82.14:4222 , piece_id=XPAUU3UNS2W5PF4XNIHW3OMLURIZKMH5RTQ6GCXHC7ABBEKZU5LA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968922361 (60h25m13.691485401s) [noise=true , address=104.36.82.15:4259 , piece_id=WLXNBS5D7IRMS6FIDGR7R3QE2OCKN7UBCPIBPXV45IUPLLMXDLJQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:53216->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968928994 (60h25m13.691492034s) [noise=true , address=104.36.82.8:4970 , piece_id=H6VCTVT6CAEW2AIUMXWJPWA4CKLG6YVRZSQF4JN5FXGLEB3PA5OQ, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:37916->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968952910 (60h25m13.69151595s) [noise=true , address=104.36.82.10:4078 , piece_id=HG4KL5XNIXMFDZGYADZ5FNPUX4TTK36UTD6SHQQSYGKSX4MONTBA, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968971746 (60h25m13.691534786s) [noise=true , address=104.36.82.2:4731 , piece_id=EXQ6KKJ3KLDNNGJ5YJY2W7AJCCBXWRWG4XQNU3KZOCITOJUAL22Q, seg_pos={0 0}, upload=57946]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:35708->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:01:03.968995992 (60h25m13.691559042s) [seg_pos={0 0}, upload=57946]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:01:04.202383707 (60h25m13.924946747s) [upload=57946]: Done uploading segments...
Jul 27 00:01:04.202415498 (60h25m13.924978538s) [upload=57946]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.235193273s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X...] duration:
12.246817958s
FAIL (2025-07-27 00:01:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.246844077s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
1.99403249s
FAIL (2025-07-27 00:00:50 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.99405258s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
19 out of 60 is failed (threshold=3) [X............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.] duration:
50.489387343s
FAIL (2025-07-27 00:00:48 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.489408945s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.] duration:
5.889514613s
FAIL (2025-07-27 00:00:46 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.889541043s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.236620041s
FAIL (2025-07-27 00:00:41 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Jul 26 23:59:42.179886117 (60h23m51.902449157s) [seg_pos={0 0}, upload=57933]: Done begin upload segment.
Jul 26 23:59:42.179910323 (60h23m51.902473363s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.179916154 (60h23m51.902479184s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.179962733 (60h23m51.902525763s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.180351267 (60h23m51.902914327s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.180808632 (60h23m51.903371672s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.180835734 (60h23m51.903398774s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181092325 (60h23m51.903655355s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181114728 (60h23m51.903677758s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181342373 (60h23m51.903905403s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181360328 (60h23m51.903923368s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181588113 (60h23m51.904151153s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.181851127 (60h23m51.904414167s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.182136052 (60h23m51.904699082s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.182437568 (60h23m51.905000598s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.182650676 (60h23m51.905213716s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.182935721 (60h23m51.905498761s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.183193584 (60h23m51.905756614s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:42.183391925 (60h23m51.905954965s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Uploading piece...
Jul 26 23:59:43.285348328 (60h23m53.007911368s) [upload=57933]: Next returned nil segment
Jul 26 23:59:43.285391911 (60h23m53.007954951s) [upload=57933]: Waiting for error group managing segments...
Jul 26 23:59:43.598226841 (60h23m53.320789891s) [noise=true , address=79.127.205.245:4225 , piece_id=GB72LCIADY3WOEQYZR7PDMYP63LPMN2ODEVEFJWB2KA2WU2D4NAA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:43.601115758 (60h23m53.323678798s) [noise=true , address=79.127.205.234:4604 , piece_id=3UTBZS2JBDUZJSL6AB2DQAASXQBXZHYDSXGYCMMS74ZQOUHIUTCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:43.642269155 (60h23m53.364832195s) [noise=true , address=64.49.13.139:4478 , piece_id=TCJFRZWBPLGWBAXUQVXLT445XZATRCZUSQQASF2F4YWZY37MC7BA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:43.668410962 (60h23m53.390974012s) [noise=true , address=64.49.13.140:4508 , piece_id=WUUEGVIHNIR6FWATY5I4VQCTVH3SUV5IGHEIZHT5LA4Z2QZJC4JA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:43.708835476 (60h23m53.431398516s) [noise=true , address=79.127.205.246:4277 , piece_id=HIP2JH62KLJDFFY3WT3WTIIKOMYGI2L5EMWROPI5DRBKDDVNRRUA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:43.718426806 (60h23m53.440989846s) [noise=true , address=64.49.13.140:4514 , piece_id=OI4KGINQDKQYEGO6OBGNJR44ZDG7RLGG25N6JULAJG6YU6YWCSGQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:44.097208810 (60h23m53.81977184s) [noise=true , address=45.91.116.10:4093 , piece_id=T3PR6RBKWHU3HDYMLDFNRAA3LM4TMVVML7G4M73BYX3D4KGUW6MA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:44.100794370 (60h23m53.823357411s) [noise=true , address=77.90.31.13:4634 , piece_id=XDHPQRFSNTYXYRSR7DAYVIUP62XY7QNAZPYUGNYN4Q77L3OFSMCA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:44.132912180 (60h23m53.85547522s) [noise=true , address=77.90.31.19:4875 , piece_id=CY73C624F2K4ZOUDKYKWQLONQJKZ2MIDYE66ZLYIVYDKK7TCDWRA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 26 23:59:44.491805251 (60h23m54.214368291s) [noise=true , address=77.90.31.15:4723 , piece_id=3HIHLPKLVBWIYAMCI52WVVR45VUHJMAXFKHW2MNZ6IPQLVKL2ODQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 27 00:00:18.017461556 (60h24m27.740024606s) [noise=true , address=77.90.31.16:4748 , piece_id=J2JZ5MJCZMNGU67WVSZ3J5ISK4PO2TGD2MIOAMMBYSP7RJX2Z67A, seg_pos={0 0}, upload=57933]: Done uploading piece. err: <nil>
Jul 27 00:00:41.095307490 (60h24m50.81787053s) [noise=true , address=104.36.82.56:4251 , piece_id=ZK5YNADTIZ7UEBJC3CB7HD6IDHI6YBIW2SPVFZBJWCDZIC2OU4VA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095364931 (60h24m50.817927971s) [noise=true , address=104.36.82.52:4093 , piece_id=RUETAUWNODUS4YTZXMNGREGOGOS4RGTRTYJ2KGY6OPC5OWG4E7YQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: rpc: dial tcp 104.36.82.52:4093: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095404997 (60h24m50.817968037s) [noise=true , address=104.36.82.46:4503 , piece_id=CPIRAHLYE3P7IS7CWKT4F6YPMFH4RTLXMIQAZHF6WKPFDPWWKKSA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:40312->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095434554 (60h24m50.817997594s) [noise=true , address=104.36.82.53:4143 , piece_id=OQSAU22KHGJVES2GEHHRL7VHQYFUV5CXXE5A6SLX3W5NZVADOMJA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095483527 (60h24m50.818046578s) [noise=true , address=104.36.82.21:4483 , piece_id=AUWEIF2JPTVPSRBGM2K7ETOBANWEES2IYGIU5CQAQVZIZIIW3NIQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:4483): protocol: rpc: dial tcp 104.36.82.21:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095546067 (60h24m50.818109107s) [noise=true , address=104.36.82.10:4091 , piece_id=3GHPE5T7NDH5U72BSYRG3O4VSNMLZPGMHTKX6DNJQ22YRUY3NXZQ, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:1yyUzkigTS9m3bDrZtAFL7tciNxpiCNswBSfBqXn1UW83GMLwi, address:104.36.82.10:4091): protocol: write tcp 188.166.225.170:53172->104.36.82.10:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095578399 (60h24m50.818141439s) [noise=true , address=77.90.31.10:4530 , piece_id=VGTF6PZXMWGHGNAST6TZ6TN24ESDVFRGUXTFT3WG25L3BFMBZWYA, seg_pos={0 0}, upload=57933]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: read tcp 188.166.225.170:48104->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:41.095609017 (60h24m50.818172057s) [seg_pos={0 0}, upload=57933]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 27 00:00:41.330560987 (60h24m51.053124027s) [upload=57933]: Done uploading segments...
Jul 27 00:00:41.330588720 (60h24m51.05315176s) [upload=57933]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.236633226s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
1.949580404s
FAIL (2025-07-27 00:00:32 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 1.949610942s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXX] duration:
30.2382672s
FAIL (2025-07-27 00:00:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
[seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments...
Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Jul 26 23:59:53.265389883 (60h24m2.987952923s) [upload=57935]: Starting upload
Jul 26 23:59:53.265451531 (60h24m2.988014571s) [upload=57935]: Done starting upload
Jul 26 23:59:53.265547215 (60h24m2.988110255s) [upload=57935]: Uploading segments...
Jul 26 23:59:53.265665881 (60h24m2.988228921s) [upload=57935]: Got next segment. Inline: false
Jul 26 23:59:53.265695618 (60h24m2.988258648s) [upload=57935]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:59:53.265700678 (60h24m2.988263718s) [upload=57935]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:59:53.500569526 (60h24m3.223132566s) [seg_pos={0 0}, upload=57935]: Begin upload segment...
Jul 26 23:59:53.500614643 (60h24m3.223177673s) [seg_pos={0 0}, upload=57935]: Done begin upload segment.
Jul 26 23:59:53.500637407 (60h24m3.223200447s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.500687723 (60h24m3.223250763s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.500991074 (60h24m3.223554114s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501094200 (60h24m3.22365724s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501236714 (60h24m3.223799744s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501365379 (60h24m3.223928419s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501482303 (60h24m3.224045343s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501645385 (60h24m3.224208425s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501744114 (60h24m3.224307154s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.501899962 (60h24m3.224463003s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502000194 (60h24m3.224563234s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502145102 (60h24m3.224708142s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502305698 (60h24m3.224868738s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502386763 (60h24m3.224949793s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502540347 (60h24m3.225103377s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:53.502817999 (60h24m3.225381039s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Uploading piece...
Jul 26 23:59:55.245077228 (60h24m4.967640258s) [upload=57935]: Next returned nil segment
Jul 26 23:59:55.245105442 (60h24m4.967668482s) [upload=57935]: Waiting for error group managing segments...
Jul 26 23:59:55.521486724 (60h24m5.244049764s) [noise=true , address=45.91.116.11:4117 , piece_id=AQBPTG5272BMUFEUV755HMOIL6ZWIEVY3ZF7T3RHNZLJNNHJV65Q, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.535621258 (60h24m5.258184288s) [noise=true , address=77.90.31.13:4628 , piece_id=7LTV6OAG4M4MDKD4LX5JO7K3ZNB65YKKJOUCVSDOEOFYQENCHLFA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.590385597 (60h24m5.312948627s) [noise=true , address=45.91.116.5:4299 , piece_id=YLVEWHEFLL5JOCOL5YQQKHN7PX2QMHTHLKPSBPTC5SUXQAHU5PPA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.596108576 (60h24m5.318671616s) [noise=true , address=45.91.116.7:4011 , piece_id=SJPUTVXVJB3UDKUWHPRRTIHSKIG4XYWMGCFMFFLG3LU574DQCGVQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.596498993 (60h24m5.319062023s) [noise=true , address=45.91.116.13:4155 , piece_id=EKJPTYCUQWZWF6ZBWYRO5ZLDBVOKYDZJGRDDWPXUJWFIJSKIDC7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.607949671 (60h24m5.330512711s) [noise=true , address=45.91.116.10:4083 , piece_id=JTOLNFY2UAAVFTK6HXUSXLUPSTIFJQ3ST2UDYOLCVKVYWH5BQSXQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.614678556 (60h24m5.337241596s) [noise=true , address=45.91.116.6:4974 , piece_id=LY4GYEJKTJFOL4DLAESIISYFOH6CIVP57MBEYFP653WAI6MSNOVA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.632027378 (60h24m5.354590419s) [noise=true , address=77.90.31.19:4841 , piece_id=FOCE3H4ENEMPV6MK4ERKDLGOT6OVJI6UXME2EO6I7IVR5TBVTAJQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.675652446 (60h24m5.398215486s) [noise=true , address=45.91.116.4:4937 , piece_id=TVXGNK4OUTL7IFRPQOT3MRNHCRIWHRSEZ6UEUCRBSPHO3ZACJMIA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 26 23:59:55.807225580 (60h24m5.529788621s) [noise=true , address=77.90.31.16:4733 , piece_id=BOS4OAPL22E5C657NRNONV2R245X5ZAS3MNYRNPF2QDMKKILOAJA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: <nil>
Jul 27 00:00:22.622487945 (60h24m32.345050985s) [noise=true , address=77.90.31.21:4878 , piece_id=ZLCYWGTEIS6SVPH5PLVTUMRHUDDLID5N6MVEJZ7LDNVZ266KNX7A, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37062->77.90.31.21:4878: use of closed network connection
Jul 27 00:00:22.622567417 (60h24m32.345130467s) [noise=true , address=104.36.82.16:4295 , piece_id=H7WWBYIN6VTSGCCCMZM7WPZXWYB5JCRFNJTUT6E6RO62PHVOBATQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622596592 (60h24m32.345159633s) [noise=true , address=104.36.82.15:4258 , piece_id=CMG2QLAEQBYMVPN4NBXPOPGTO5TBA6IXC22IOROJIKBCBAQKYEKA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622627972 (60h24m32.345191012s) [noise=true , address=104.36.82.5:4840 , piece_id=3FHF4GE74SVX2L2NBJ27PYZOHQLZR45R5IKVDV5IRLTYVGFI4KHQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622653822 (60h24m32.345216862s) [noise=true , address=104.36.82.4:4803 , piece_id=NEFEOBIL2O55TE2KVV5IDFNXN2C35WCV34RDBJENQFQGYEEN3FWQ, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622694220 (60h24m32.34525726s) [noise=true , address=104.36.82.12:4151 , piece_id=WFN4GWDCS5RK7KJCPAQS5FYQLQYMVHAZW6OX6QLZOLEK3UAVBREA, seg_pos={0 0}, upload=57935]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:53968->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:00:22.622712734 (60h24m32.345275785s) [seg_pos={0 0}, upload=57935]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:00:22.858870780 (60h24m32.58143383s) [upload=57935]: Done uploading segments...
Jul 27 00:00:22.858903663 (60h24m32.581466713s) [upload=57935]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.238292498s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X..] duration:
5.917415052s
FAIL (2025-07-27 00:00:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.917451352s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX] duration:
30.244538304s
FAIL (2025-07-27 00:00:08 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.244561219s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [XX............XXXX.....XXX..X.....X......X...XX..XX.X.XX...X] duration:
1m0.00115051s
FAIL (2025-07-26 23:59:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001161703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXX] duration:
30.239655162s
FAIL (2025-07-26 23:59:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
6 23:59:18.755564131 (60h23m28.478127171s) [upload=57925]: Starting upload
Jul 26 23:59:18.755617964 (60h23m28.478181004s) [upload=57925]: Done starting upload
Jul 26 23:59:18.755706744 (60h23m28.478269775s) [upload=57925]: Uploading segments...
Jul 26 23:59:18.755814431 (60h23m28.478377471s) [upload=57925]: Got next segment. Inline: false
Jul 26 23:59:18.755857272 (60h23m28.478420312s) [upload=57925]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 26 23:59:18.755863173 (60h23m28.478426223s) [upload=57925]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 26 23:59:18.989710817 (60h23m28.712273857s) [seg_pos={0 0}, upload=57925]: Begin upload segment...
Jul 26 23:59:18.989748790 (60h23m28.71231183s) [seg_pos={0 0}, upload=57925]: Done begin upload segment.
Jul 26 23:59:18.989766403 (60h23m28.712329443s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.989850995 (60h23m28.712414035s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.990095784 (60h23m28.712658824s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.990392171 (60h23m28.712955211s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.990510397 (60h23m28.713073437s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.990643101 (60h23m28.713206131s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.990975377 (60h23m28.713538417s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991103472 (60h23m28.713666542s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991392835 (60h23m28.713955875s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991486916 (60h23m28.714049956s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991739719 (60h23m28.714302749s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991790737 (60h23m28.714353777s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.991946946 (60h23m28.714509986s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.992186614 (60h23m28.714749644s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.992254975 (60h23m28.714818015s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:18.992672093 (60h23m28.715235133s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Uploading piece...
Jul 26 23:59:20.729860921 (60h23m30.452423961s) [upload=57925]: Next returned nil segment
Jul 26 23:59:20.729875970 (60h23m30.45243901s) [upload=57925]: Waiting for error group managing segments...
Jul 26 23:59:21.008316574 (60h23m30.730879614s) [noise=true , address=45.91.116.5:4297 , piece_id=PN5DV2QIFSNJRCCBDXHMP6FRE53QF2HYSJB6KACF2D25BRVI43GQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:21.008638079 (60h23m30.731201119s) [noise=true , address=45.91.116.9:4263 , piece_id=LRTRZL5BVH5QCFALUX33SJJKE24WDQZ2OAVWR553AN4RKTUQX6YQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:21.024163012 (60h23m30.746726052s) [noise=true , address=77.90.31.18:4807 , piece_id=JD44JYPSAUIRVUVR335BYRNOO7YKK6LNQHRFT3FHGPX7YPK5TNIA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:21.160487698 (60h23m30.883050738s) [noise=true , address=45.91.116.4:4937 , piece_id=7LROMNYY4KI5ZU5UUIFWFAKHRBPY3ZX4MKD6BDGSSETRS4P62HLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:21.162286621 (60h23m30.884849661s) [noise=true , address=45.91.116.12:4225 , piece_id=XAKZZ5M4VQTXRW2OSERVJNGEFGLYHOJFUDCLM3EYHXKEGSA3V5VQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:21.461775184 (60h23m31.184338224s) [noise=true , address=77.90.31.19:4842 , piece_id=RBA2YJT7GTFTY27IV6TPA3WYF3LYHHCAG4SQJYS6CX5KVX5XDC4Q, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:22.102625777 (60h23m31.825188817s) [noise=true , address=77.90.31.11:4553 , piece_id=FDCDUZQMRFAFO26V2VWR2K2C4FMHYMPIYKTURD4RWRTBQ77DCTPA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:22.398677264 (60h23m32.121240304s) [noise=true , address=77.90.31.15:4699 , piece_id=TTBER3ELYLVLJRQGYNDXYMI3JVHZ4QXTVWYKE325MOM3RN5KMPIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: <nil>
Jul 26 23:59:48.106089545 (60h23m57.828652585s) [noise=true , address=77.90.31.17:4771 , piece_id=X2MWNQBSQDBZT7IAORHT37I3OBI7NCSNIKI73I4DR4DOKI5OVXLQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106139840 (60h23m57.82870288s) [noise=true , address=77.90.31.21:4877 , piece_id=D2SXNWMSP45DREWPOVS2ZPEFQ4AUQPGVHIWS4I2MZAH6CAJWBPRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: rpc: dial tcp 77.90.31.21:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106225273 (60h23m57.828788313s) [noise=true , address=104.36.82.7:4909 , piece_id=42TCPNK4VWENSWTDGQOYACAK3OH3EPB27VENMKGHKFMZV63VMVRQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:35526->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106281591 (60h23m57.828844631s) [noise=true , address=104.36.82.8:4972 , piece_id=6HCX6NPRHFQFA5N6FM7INKKWF7UM5W6T47YVWRZWAISWA2V4QGAQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106335895 (60h23m57.828898925s) [noise=true , address=104.36.82.6:4875 , piece_id=F6SVR5XUG4NQRYDEOI63SEFN4B5L7OGDXJZFN5FY4DAVV53ZSXIQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106366715 (60h23m57.828929755s) [noise=true , address=104.36.82.7:4911 , piece_id=XKCXZ7ZI5MBLV2BVCFI5S3WC5IR62LSUJC6X7LWYOJGYEQNWI6EA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:42844->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106414085 (60h23m57.828977115s) [noise=true , address=104.36.82.10:4079 , piece_id=X43FAYVRYEAKS77WBNOWTOTCWQ4QJQLW4COWUP4SDF6ZXFVC7IQA, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106446386 (60h23m57.829009426s) [noise=true , address=104.36.82.2:4732 , piece_id=LFO622WQ6TM74VHXWI2KMY5QOKJTNRDE2O55CPT5YHBSHLFBUQDQ, seg_pos={0 0}, upload=57925]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:48.106468108 (60h23m57.829031148s) [seg_pos={0 0}, upload=57925]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 26 23:59:48.343570641 (60h23m58.066133681s) [upload=57925]: Done uploading segments...
Jul 26 23:59:48.343600899 (60h23m58.066163989s) [upload=57925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.239673657s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.233995905s
FAIL (2025-07-26 23:59:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.727576947 (60h22m51.450139977s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.728137028 (60h22m51.450700068s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.728432825 (60h22m51.450995865s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.728479454 (60h22m51.451042494s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.728761574 (60h22m51.451324614s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.728956907 (60h22m51.451519947s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.729045346 (60h22m51.451608386s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.729528752 (60h22m51.452091802s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.729779631 (60h22m51.452342661s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.730008299 (60h22m51.452571339s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.730341586 (60h22m51.452904626s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.730668923 (60h22m51.453231963s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.731030434 (60h22m51.453593474s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.731448755 (60h22m51.454011785s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.732112174 (60h22m51.454675214s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.732720969 (60h22m51.455284019s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:41.733091287 (60h22m51.455654327s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Uploading piece...
Jul 26 23:58:42.838834658 (60h22m52.561397688s) [upload=57914]: Next returned nil segment
Jul 26 23:58:42.838851310 (60h22m52.56141435s) [upload=57914]: Waiting for error group managing segments...
Jul 26 23:58:43.129009261 (60h22m52.851572311s) [noise=true , address=79.127.205.234:4589 , piece_id=TBK5GNMYMDQ7LISJKSTYNXVGRMZU5T4ZJFR2OAEW7PWVJ2MVNYYQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:43.138456927 (60h22m52.861019957s) [noise=true , address=79.127.205.246:4281 , piece_id=GBOBLPDXPS2B5Q72YD7CSAOT3HTUYFVPR55POSDG53XAWRGGXZQQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:43.163842056 (60h22m52.886405096s) [noise=true , address=79.127.205.235:4689 , piece_id=QDQ4RA6PYGZBGCZ2VT3AHNE76K7PLCYCPXRP3BGVEJENFLJ2RQYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:43.583641918 (60h22m53.306204958s) [noise=true , address=79.127.220.101:4041 , piece_id=QDEXNFNLMPTFETNMYHPXSP4YEJNYKFTKOTG32MQLW3JJM543WJNQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:43.724061759 (60h22m53.446624799s) [noise=true , address=79.127.219.38:4010 , piece_id=XE3WNW4V2GGMPM5RMFQVKN3WYLDVZ5C5QFQQZPSNE7XSJ77QHMEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:43.801955510 (60h22m53.52451855s) [noise=true , address=79.127.219.48:4074 , piece_id=QDGK25JTUZI2X4QES4IOKWXUJLSRQU67L45SWSPSLWJOWEU4HETA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:44.316271165 (60h22m54.038834246s) [noise=true , address=79.127.219.47:4048 , piece_id=OH4LGS3EXOQTPSNFJHHXWTGBLLC3WLNRXF5C72LOHKVOGW3N23PA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:58:45.515232647 (60h22m55.237795707s) [noise=true , address=79.127.220.98:4422 , piece_id=HC25PY5R7QA7B5Z2IP7UYQLJX5C7V3BSMOEK7IJ2TIRYOI6DRMKQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:59:08.772393964 (60h23m18.494957004s) [noise=true , address=77.90.31.19:4871 , piece_id=LVSLGJUM2RAGRVWXNED3HYLLEGWIIV6BJPK3M522NXU4W56LQJCQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:59:23.999032802 (60h23m33.721595842s) [noise=true , address=79.127.201.215:4354 , piece_id=3JQGNQSPTKDPMPO2OBVYURKA6VINIFHGA3OUHITWAND6R6B3O4SQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: <nil>
Jul 26 23:59:40.631535807 (60h23m50.354098847s) [noise=true , address=104.36.82.47:4534 , piece_id=GJBJ7K2YRVVRVOPZUAZCZ4GPSZ3THYFYPW3QZ3M446J35WX3KFEA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: rpc: dial tcp 104.36.82.47:4534: i/o timeout; conn closed: op
Jul 26 23:59:40.631613436 (60h23m50.354176476s) [noise=true , address=104.36.82.55:4213 , piece_id=VPPAFN7667ZDCIW675V4U7SJLPG5UCODOUFDAIT2UNQLRKEYTN3Q, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631773583 (60h23m50.354336623s) [noise=true , address=104.36.82.53:4144 , piece_id=QDFKFHHS3T345EHALSLNMQANV4YKCCOZWTFVT6BMJBCMZ7MN4TIA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1FfzcPNLmNdrvoZBKcmCLz8d9rYSAnjFTo4NBnxBzN6dVcTNwt, address:104.36.82.53:4144): protocol: write tcp 188.166.225.170:58646->104.36.82.53:4144: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631808780 (60h23m50.35437182s) [noise=true , address=77.90.31.10:4533 , piece_id=4XVWMSL6FFG3PQGFV3HKJDKWGGZXHJ3VZ67P2O4UZV7IVG3SVXMQ, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: read tcp 188.166.225.170:44574->77.90.31.10:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631841593 (60h23m50.354404633s) [noise=true , address=104.36.82.44:4870 , piece_id=2WZ44IFYNHMMXG5TA34SCXHTZ2OXXW2MBW5WVHFDTGK4UCBYPOYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: write tcp 188.166.225.170:40786->104.36.82.44:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631884745 (60h23m50.354447785s) [noise=true , address=104.36.82.56:4238 , piece_id=PVPM3FKXUAD53XMG5XJSDNXZL7TF7LKSDZAD3KT73NEKDOHWWSWA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: rpc: dial tcp 104.36.82.56:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631949940 (60h23m50.35451298s) [noise=true , address=104.36.82.52:4092 , piece_id=5R2D3PAYP72ELO3R4XRG2W3PSWXVRNEBOTJPXQGNCHAWOOGXPDRA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 188.166.225.170:59158->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.631997791 (60h23m50.354560821s) [noise=true , address=104.36.82.40:4942 , piece_id=YXTRNOSDKYED3W4NK44W327XBDKCVSVGAWTTBDJAX4ZCZO7RDGYA, seg_pos={0 0}, upload=57914]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 26 23:59:40.632018481 (60h23m50.354581521s) [seg_pos={0 0}, upload=57914]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Jul 26 23:59:40.863499046 (60h23m50.586062086s) [upload=57914]: Done uploading segments...
Jul 26 23:59:40.863526769 (60h23m50.586089799s) [upload=57914]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.234016886s
core
uplink
us-select-1
us1-placement-20
|