Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X........X.......................] duration:
24.048806621s
FAIL (2025-07-09 19:43:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.048821879s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
4.839437552s
FAIL (2025-07-09 19:43:35 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.83946214s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration:
19.177932119s
FAIL (2025-07-09 19:43:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 19.177958147s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X...X......................................................] duration:
29.483966864s
FAIL (2025-07-09 19:43:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.483989789s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
4.795877158s
FAIL (2025-07-09 19:43:16 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.795895813s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration:
27.029451503s
FAIL (2025-07-09 19:42:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.029472842s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
30.860945758s
FAIL (2025-07-09 19:42:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 30.860974874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X........X......................] duration:
5.86470857s
FAIL (2025-07-09 19:42:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.864779556s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
4.855077286s
FAIL (2025-07-09 19:42:11 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.855102645s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X...X....................................................] duration:
17.548135449s
FAIL (2025-07-09 19:41:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.5481572s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X........X.....................] duration:
6.30284249s
FAIL (2025-07-09 19:41:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.302864883s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X...X...................................................] duration:
40.759065248s
FAIL (2025-07-09 19:41:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 40.759087671s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
5.899073767s
FAIL (2025-07-09 19:41:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 5.89909139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X..X] duration:
1m0.231908953s
FAIL (2025-07-09 19:41:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
[upload=315633]: Got next segment. Inline: false
Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment...
Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment.
Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece...
Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000
Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment
Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments...
Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil>
Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF
Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection
Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments...
Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.231935313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
4.898048105s
FAIL (2025-07-09 19:40:57 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.898085407s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X........X....................] duration:
6.06401843s
FAIL (2025-07-09 19:40:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.064040782s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X........X...................] duration:
6.545945519s
FAIL (2025-07-09 19:40:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.545986587s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
4.748199455s
FAIL (2025-07-09 19:40:16 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.74822807s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration:
38.948234019s
FAIL (2025-07-09 19:39:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.948254539s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
36.106371727s
FAIL (2025-07-09 19:39:36 UTC)
orbiter warning:
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
upload
Jul 9 19:37:04.754477644 (394h26m39.525118969s) [upload=315597]: Done starting upload
Jul 9 19:37:04.754539242 (394h26m39.525180577s) [upload=315597]: Uploading segments...
Jul 9 19:37:04.754689439 (394h26m39.525330774s) [upload=315597]: Got next segment. Inline: false
Jul 9 19:37:04.754731780 (394h26m39.525373115s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:37:04.754736960 (394h26m39.525378295s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:37:05.211923957 (394h26m39.982565292s) [seg_pos={0 0}, upload=315597]: Begin upload segment...
Jul 9 19:37:05.211975275 (394h26m39.98261661s) [seg_pos={0 0}, upload=315597]: Done begin upload segment.
Jul 9 19:37:05.212024600 (394h26m39.982665935s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212031523 (394h26m39.982672858s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212381212 (394h26m39.983022547s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212488517 (394h26m39.983129852s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212695272 (394h26m39.983336607s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213067014 (394h26m39.983708339s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213376766 (394h26m39.984018102s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213603881 (394h26m39.984245216s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214102815 (394h26m39.98474415s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214223276 (394h26m39.984864611s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214493714 (394h26m39.985135039s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214743241 (394h26m39.985384576s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215511440 (394h26m39.986152765s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215877371 (394h26m39.986518696s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.222557812 (394h26m39.993199147s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.224738312 (394h26m39.995379637s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.226179941 (394h26m39.996821276s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.227318920 (394h26m39.997960255s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.982931871 (394h26m40.753573206s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: ecclient: upload failed (node:19w7rcXTHunzTLofD6WRyUH9yTc9nWDGCWBYBi5UpxdKVCSk7A, address:79.127.205.233:4626): protocol: storage node overloaded, request limit: 2000
Jul 9 19:37:06.652009436 (394h26m41.422650771s) [upload=315597]: Next returned nil segment
Jul 9 19:37:06.652026708 (394h26m41.422668033s) [upload=315597]: Waiting for error group managing segments...
Jul 9 19:37:07.203506280 (394h26m41.974147616s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.205635194 (394h26m41.976276529s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.213846744 (394h26m41.984488069s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217355106 (394h26m41.987996431s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217595736 (394h26m41.988237071s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217954533 (394h26m41.988595868s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.225783310 (394h26m41.996424645s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.229734178 (394h26m42.000375503s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.239927921 (394h26m42.010569256s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:12.064948314 (394h26m46.835589639s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:25.245409234 (394h27m0.016050599s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:40.715040429 (394h27m15.485681764s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:41.811994636 (394h27m16.582635981s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064578202 (394h27m35.835219537s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064607918 (394h27m35.835249253s) [seg_pos={0 0}, upload=315597]: Segment reached optimal threshold of 14 pieces.
Jul 9 19:38:01.064711807 (394h27m35.835353142s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064758997 (394h27m35.835400332s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064791650 (394h27m35.835432975s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064828280 (394h27m35.835469615s) [seg_pos={0 0}, upload=315597]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 19:38:01.064906119 (394h27m35.835547454s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 19:38:01.064911179 (394h27m35.835552514s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 19:38:01.371186080 (394h27m36.141827415s) [upload=315597]: Done uploading segments...
Jul 9 19:38:01.371207982 (394h27m36.141849317s) [upload=315597]: Upload finished. err: <nil>
Duration: 36.106402195s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X........X..................] duration:
6.062326091s
FAIL (2025-07-09 19:39:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.062347212s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......X...X.................................................] duration:
29.985291653s
FAIL (2025-07-09 19:39:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.985313584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
4.691492494s
FAIL (2025-07-09 19:39:06 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.691522091s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X........X.................] duration:
6.678027725s
FAIL (2025-07-09 19:38:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.678055508s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
4.78131135s
FAIL (2025-07-09 19:38:24 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.781337219s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
18.371695608s
FAIL (2025-07-09 19:38:22 UTC)
orbiter warning:
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
upload
Jul 9 19:37:04.754477644 (394h26m39.525118969s) [upload=315597]: Done starting upload
Jul 9 19:37:04.754539242 (394h26m39.525180577s) [upload=315597]: Uploading segments...
Jul 9 19:37:04.754689439 (394h26m39.525330774s) [upload=315597]: Got next segment. Inline: false
Jul 9 19:37:04.754731780 (394h26m39.525373115s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:37:04.754736960 (394h26m39.525378295s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:37:05.211923957 (394h26m39.982565292s) [seg_pos={0 0}, upload=315597]: Begin upload segment...
Jul 9 19:37:05.211975275 (394h26m39.98261661s) [seg_pos={0 0}, upload=315597]: Done begin upload segment.
Jul 9 19:37:05.212024600 (394h26m39.982665935s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212031523 (394h26m39.982672858s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212381212 (394h26m39.983022547s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212488517 (394h26m39.983129852s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212695272 (394h26m39.983336607s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213067014 (394h26m39.983708339s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213376766 (394h26m39.984018102s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213603881 (394h26m39.984245216s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214102815 (394h26m39.98474415s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214223276 (394h26m39.984864611s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214493714 (394h26m39.985135039s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214743241 (394h26m39.985384576s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215511440 (394h26m39.986152765s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215877371 (394h26m39.986518696s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.222557812 (394h26m39.993199147s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.224738312 (394h26m39.995379637s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.226179941 (394h26m39.996821276s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.227318920 (394h26m39.997960255s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.982931871 (394h26m40.753573206s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: ecclient: upload failed (node:19w7rcXTHunzTLofD6WRyUH9yTc9nWDGCWBYBi5UpxdKVCSk7A, address:79.127.205.233:4626): protocol: storage node overloaded, request limit: 2000
Jul 9 19:37:06.652009436 (394h26m41.422650771s) [upload=315597]: Next returned nil segment
Jul 9 19:37:06.652026708 (394h26m41.422668033s) [upload=315597]: Waiting for error group managing segments...
Jul 9 19:37:07.203506280 (394h26m41.974147616s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.205635194 (394h26m41.976276529s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.213846744 (394h26m41.984488069s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217355106 (394h26m41.987996431s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217595736 (394h26m41.988237071s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217954533 (394h26m41.988595868s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.225783310 (394h26m41.996424645s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.229734178 (394h26m42.000375503s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.239927921 (394h26m42.010569256s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:12.064948314 (394h26m46.835589639s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:25.245409234 (394h27m0.016050599s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:40.715040429 (394h27m15.485681764s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:41.811994636 (394h27m16.582635981s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064578202 (394h27m35.835219537s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064607918 (394h27m35.835249253s) [seg_pos={0 0}, upload=315597]: Segment reached optimal threshold of 14 pieces.
Jul 9 19:38:01.064711807 (394h27m35.835353142s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064758997 (394h27m35.835400332s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064791650 (394h27m35.835432975s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064828280 (394h27m35.835469615s) [seg_pos={0 0}, upload=315597]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 19:38:01.064906119 (394h27m35.835547454s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 19:38:01.064911179 (394h27m35.835552514s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 19:38:01.371186080 (394h27m36.141827415s) [upload=315597]: Done uploading segments...
Jul 9 19:38:01.371207982 (394h27m36.141849317s) [upload=315597]: Upload finished. err: <nil>
Duration: 18.371715937s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.002345514s
FAIL (2025-07-09 19:38:04 UTC)
orbiter warning:
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
upload
Jul 9 19:37:04.754477644 (394h26m39.525118969s) [upload=315597]: Done starting upload
Jul 9 19:37:04.754539242 (394h26m39.525180577s) [upload=315597]: Uploading segments...
Jul 9 19:37:04.754689439 (394h26m39.525330774s) [upload=315597]: Got next segment. Inline: false
Jul 9 19:37:04.754731780 (394h26m39.525373115s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 19:37:04.754736960 (394h26m39.525378295s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 19:37:05.211923957 (394h26m39.982565292s) [seg_pos={0 0}, upload=315597]: Begin upload segment...
Jul 9 19:37:05.211975275 (394h26m39.98261661s) [seg_pos={0 0}, upload=315597]: Done begin upload segment.
Jul 9 19:37:05.212024600 (394h26m39.982665935s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212031523 (394h26m39.982672858s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212381212 (394h26m39.983022547s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212488517 (394h26m39.983129852s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.212695272 (394h26m39.983336607s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213067014 (394h26m39.983708339s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213376766 (394h26m39.984018102s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.213603881 (394h26m39.984245216s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214102815 (394h26m39.98474415s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214223276 (394h26m39.984864611s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214493714 (394h26m39.985135039s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.214743241 (394h26m39.985384576s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215511440 (394h26m39.986152765s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.215877371 (394h26m39.986518696s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.222557812 (394h26m39.993199147s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.224738312 (394h26m39.995379637s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.226179941 (394h26m39.996821276s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.227318920 (394h26m39.997960255s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Uploading piece...
Jul 9 19:37:05.982931871 (394h26m40.753573206s) [noise=true , address=79.127.205.233:4626 , piece_id=K5KOSI6XPZSC4QMA7L2TVAEDIBFJ6SEDJL4BBQZGMRDDCB4TQGWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: ecclient: upload failed (node:19w7rcXTHunzTLofD6WRyUH9yTc9nWDGCWBYBi5UpxdKVCSk7A, address:79.127.205.233:4626): protocol: storage node overloaded, request limit: 2000
Jul 9 19:37:06.652009436 (394h26m41.422650771s) [upload=315597]: Next returned nil segment
Jul 9 19:37:06.652026708 (394h26m41.422668033s) [upload=315597]: Waiting for error group managing segments...
Jul 9 19:37:07.203506280 (394h26m41.974147616s) [noise=true , address=77.90.31.10:4529 , piece_id=M7SVRXFH2RSCVCXFYJNSYJ2HGWLCLJO5MMMKBZLMPRJSKPHKZF3Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.205635194 (394h26m41.976276529s) [noise=true , address=77.90.31.15:4717 , piece_id=KOQX63X7EKHIKDOPIFUX7SI7C3Z7KC5WMGG2ZMWVKRUEB74PP3KA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.213846744 (394h26m41.984488069s) [noise=true , address=77.90.31.16:4767 , piece_id=2E6SGLYEECFHS36MGXXAWOBLRNZRWQNHBNQMPN4HJLX7K7CJYLWQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217355106 (394h26m41.987996431s) [noise=true , address=77.90.31.14:4668 , piece_id=VCHHRVMCSEUPGCDTK6U2NT5WYBW4BPL3TK7ZDYL3RDQ6BGRJO4IQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217595736 (394h26m41.988237071s) [noise=true , address=77.90.31.21:4883 , piece_id=HLEOSU2B2EW3WCQELYILDRCZKKVXCOLCGYD3D4AANQXJYMCF4ZAQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.217954533 (394h26m41.988595868s) [noise=true , address=77.90.31.12:4610 , piece_id=Z3NV4WSYR2WC6D7HLZRHFFQ2UPFNLJK2DN2KS7MQV63HN5QFEDRA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.225783310 (394h26m41.996424645s) [noise=true , address=77.90.31.11:4570 , piece_id=57WTCETYAJ4WIHVNCFW442CJDVUK2ECQOHBF7NLZ76ON7ATPE6FQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.229734178 (394h26m42.000375503s) [noise=true , address=77.90.31.17:4801 , piece_id=YVXLKOZANBSUBRDL6F6TWC7PLMK3L7QW4JD6B7YVTWISMLNJ4S6Q, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:07.239927921 (394h26m42.010569256s) [noise=true , address=77.90.31.19:4868 , piece_id=Q6KM6JTLSYJNFS2BAM7UDGLOMEGSJHJ5UDVGFYUEXYQXLRJE4NTA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:12.064948314 (394h26m46.835589639s) [noise=true , address=79.127.220.98:4417 , piece_id=CYXBLEXTLBBNORUXU2DABPQ25ZEO5ODBYQS3UDXUQARWVV5LHJQA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:25.245409234 (394h27m0.016050599s) [noise=true , address=104.36.82.34:4570 , piece_id=KFSKZTXRBELLXMGPVVDWB4NG4RSTCEW4CTUGKHKWWJXML5L7G2PA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:40.715040429 (394h27m15.485681764s) [noise=true , address=104.36.82.8:4983 , piece_id=A3LJGTKNRBO36EBCO4CR4NL25GNWJNSF7UXU4AFX6XQJRVQH2BJA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:37:41.811994636 (394h27m16.582635981s) [noise=true , address=104.36.82.35:4585 , piece_id=76CMW3WQWIQZYUJZQIAPZ4KKADAKBFRMUA72ZS2GEVG5RCWKGN2A, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064578202 (394h27m35.835219537s) [noise=true , address=104.36.82.22:4522 , piece_id=K2HGHSABQGTGP2CAVWBTMXTQEG3ZOPTMD4PTVUM5QREW7QU4XHIA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: <nil>
Jul 9 19:38:01.064607918 (394h27m35.835249253s) [seg_pos={0 0}, upload=315597]: Segment reached optimal threshold of 14 pieces.
Jul 9 19:38:01.064711807 (394h27m35.835353142s) [noise=true , address=104.36.82.36:4642 , piece_id=UMMJCC63Z6UC5U6WDD6DRCRIX432PK2KVZ6PWI4UYZF3VJZRA6QA, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064758997 (394h27m35.835400332s) [noise=true , address=104.36.82.44:4903 , piece_id=LLK3ZHW2I3OWD33C5GWNBNOIDXFK3XMQICAQIK45456X3G2GGZUQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064791650 (394h27m35.835432975s) [noise=true , address=104.36.82.18:4383 , piece_id=RXAZQNGZDMS4R5WYLRJNQH2PBFWGQSPVIQAE56H5GGV546KAFTPQ, seg_pos={0 0}, upload=315597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Wtu7qHz3z5gYVPLJzaKKH3orBrtBUEVUeHRUaUe4xJGZv4u4z): protocol: expected piece hash; context canceled; EOF
Jul 9 19:38:01.064828280 (394h27m35.835469615s) [seg_pos={0 0}, upload=315597]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 19:38:01.064906119 (394h27m35.835547454s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 19:38:01.064911179 (394h27m35.835552514s) [upload=315597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 19:38:01.371186080 (394h27m36.141827415s) [upload=315597]: Done uploading segments...
Jul 9 19:38:01.371207982 (394h27m36.141849317s) [upload=315597]: Upload finished. err: <nil>
Duration: 1m0.002363499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X......X...X................................................] duration:
24.02043771s
FAIL (2025-07-09 19:37:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.020459511s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
4.843321434s
FAIL (2025-07-09 19:37:46 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.843347554s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X........X................] duration:
6.219169004s
FAIL (2025-07-09 19:37:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.219192391s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X........X...............] duration:
6.128103438s
FAIL (2025-07-09 19:37:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.128122284s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.059411199s
FAIL (2025-07-09 19:37:08 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 5.059438379s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X......X...X...............................................] duration:
26.157936856s
FAIL (2025-07-09 19:37:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.157961121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X......X...X..............................................] duration:
23.717592035s
FAIL (2025-07-09 19:36:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.717617685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X........X..............] duration:
6.274109538s
FAIL (2025-07-09 19:36:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.274128193s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
4.691626761s
FAIL (2025-07-09 19:36:24 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.691651479s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X........X.............] duration:
6.14904546s
FAIL (2025-07-09 19:35:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.14907163s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X......X...X.............................................] duration:
22.877968871s
FAIL (2025-07-09 19:35:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.877990332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X........X............] duration:
6.451382327s
FAIL (2025-07-09 19:35:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.451403167s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.117211568s
FAIL (2025-07-09 19:35:15 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 5.117237698s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration:
6.190551568s
FAIL (2025-07-09 19:35:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.190599691s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X......X...X............................................] duration:
31.004915984s
FAIL (2025-07-09 19:34:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.004947543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X........X..........] duration:
5.998015171s
FAIL (2025-07-09 19:34:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.99820792s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
4.933637704s
FAIL (2025-07-09 19:34:06 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.933665847s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration:
6.028438611s
FAIL (2025-07-09 19:34:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.028456907s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X......X...X...........................................] duration:
31.502661181s
FAIL (2025-07-09 19:33:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.502683895s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X........X........] duration:
6.097321596s
FAIL (2025-07-09 19:33:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.097350872s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration:
6.030485249s
FAIL (2025-07-09 19:33:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.030512061s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
4.886722198s
FAIL (2025-07-09 19:33:07 UTC)
orbiter warning: uplink: metaclient: context deadline exceeded
Duration: 4.886782233s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X......X...X..........................................] duration:
23.752778074s
FAIL (2025-07-09 19:32:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.752790266s
core
uplink
us-select-1
us1-placement-20
|