Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 331610 reports (Page 6323 of 6633)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.........X..X..] duration: 24.635636001s FAIL (2025-07-09 19:57:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.635649376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X........X.......................X.................] duration: 5.916388191s FAIL (2025-07-09 19:57:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.916410515s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.........X..X.] duration: 41.957591697s FAIL (2025-07-09 19:57:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.95761877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.952862674s FAIL (2025-07-09 19:57:02 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.952887812s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X........X.......................X................] duration: 6.447963095s FAIL (2025-07-09 19:56:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.447997321s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 4.735870556s FAIL (2025-07-09 19:56:39 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.735903519s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X........X.......................X...............] duration: 6.407130574s FAIL (2025-07-09 19:56:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.407159599s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration: 23.934602449s FAIL (2025-07-09 19:56:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.934655249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.........X..X] duration: 1m0.000849304s FAIL (2025-07-09 19:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000861127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.70733108s FAIL (2025-07-09 19:56:04 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.707359605s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..X................] duration: 31.884238106s FAIL (2025-07-09 19:55:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.884269295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X........X.......................X..............] duration: 5.897682805s FAIL (2025-07-09 19:55:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.897705448s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration: 47.436972591s FAIL (2025-07-09 19:55:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.436998923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.118421661s FAIL (2025-07-09 19:55:07 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 5.118453513s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X........X.......................X.............] duration: 6.53802575s FAIL (2025-07-09 19:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.538047022s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.924847706s FAIL (2025-07-09 19:54:23 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.924869016s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X........X.......................X............] duration: 6.039155422s FAIL (2025-07-09 19:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.039179628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration: 26.859719406s FAIL (2025-07-09 19:54:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.85974218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.........X.] duration: 28.241298032s FAIL (2025-07-09 19:54:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.241445084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X........X.......................X...........] duration: 6.452517068s FAIL (2025-07-09 19:53:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.452539681s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.020637296s FAIL (2025-07-09 19:53:57 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 5.020666874s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.714712175s FAIL (2025-07-09 19:53:34 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.714738596s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.........X] duration: 1m0.004458726s FAIL (2025-07-09 19:53:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.004470129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration: 8.726866157s FAIL (2025-07-09 19:53:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.72688834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X........X.......................X..........] duration: 6.217995723s FAIL (2025-07-09 19:52:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.218019288s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.686219916s FAIL (2025-07-09 19:52:52 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.686247378s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration: 25.973002298s FAIL (2025-07-09 19:52:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.973019401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X........X.......................X.........] duration: 6.919532443s FAIL (2025-07-09 19:52:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.919550046s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..X............] duration: 28.526681868s FAIL (2025-07-09 19:52:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.526724039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.838886947s FAIL (2025-07-09 19:52:08 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.83890954s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X........X.......................X........] duration: 6.376542269s FAIL (2025-07-09 19:51:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.376562105s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 33.289390752s FAIL (2025-07-09 19:51:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.289413796s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.095176895s FAIL (2025-07-09 19:51:45 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 5.095203106s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration: 26.022215136s FAIL (2025-07-09 19:51:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.02224307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.735507098s FAIL (2025-07-09 19:51:12 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.735536193s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 46.927222092s FAIL (2025-07-09 19:51:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.927246189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X........X.......................X.......] duration: 6.736925714s FAIL (2025-07-09 19:51:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.736947288s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.121671721s FAIL (2025-07-09 19:50:49 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 5.121698021s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration: 18.987438219s FAIL (2025-07-09 19:50:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.987468105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.65643893s FAIL (2025-07-09 19:50:22 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.656465802s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X........X.......................X......] duration: 6.291811395s FAIL (2025-07-09 19:50:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.291831353s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.103970525s FAIL (2025-07-09 19:50:07 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 5.104003419s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 19.625005429s FAIL (2025-07-09 19:49:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.625019436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 36.354999817s FAIL (2025-07-09 19:49:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.355013714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X........X.......................X.....] duration: 6.299070905s FAIL (2025-07-09 19:49:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.299095502s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration: 5.721696859s FAIL (2025-07-09 19:49:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.721711908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.533382211s FAIL (2025-07-09 19:48:59 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 4.533403902s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..X........] duration: 48.458392657s FAIL (2025-07-09 19:48:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== [upload=315633]: Got next segment. Inline: false Jul 9 19:40:04.519820447 (394h29m39.290461792s) [upload=315633]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 19:40:04.519828232 (394h29m39.290469577s) [upload=315633]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 19:40:04.965170358 (394h29m39.735811693s) [seg_pos={0 0}, upload=315633]: Begin upload segment... Jul 9 19:40:04.965222858 (394h29m39.735864193s) [seg_pos={0 0}, upload=315633]: Done begin upload segment. Jul 9 19:40:04.965238298 (394h29m39.735879633s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965269007 (394h29m39.735910342s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965654054 (394h29m39.736295389s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965663902 (394h29m39.736305237s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.965939379 (394h29m39.736580714s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966213023 (394h29m39.736854358s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966457551 (394h29m39.737098886s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966712358 (394h29m39.737353693s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966748267 (394h29m39.737389602s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.966978357 (394h29m39.737619692s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967191916 (394h29m39.737833251s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967408260 (394h29m39.738049595s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967515284 (394h29m39.738156619s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.967778298 (394h29m39.738419633s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968025170 (394h29m39.738666505s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968250731 (394h29m39.738892066s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968337527 (394h29m39.738978872s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:04.968730469 (394h29m39.739371804s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Uploading piece... Jul 9 19:40:05.941879859 (394h29m40.712521194s) [noise=true , address=79.127.220.98:4417 , piece_id=TBFFSVCJEIODOHDOU2QWZROYVWIVVJ7F6HWQPT4DWET4SOCFOKJQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:124CZz86euKySugctTJCwUePTumkgqktxtJY7E47aEXLwuqURTj, address:79.127.220.98:4417): protocol: storage node overloaded, request limit: 2000 Jul 9 19:40:06.468687360 (394h29m41.239328715s) [upload=315633]: Next returned nil segment Jul 9 19:40:06.468714853 (394h29m41.239356188s) [upload=315633]: Waiting for error group managing segments... Jul 9 19:40:06.952895733 (394h29m41.723537068s) [noise=true , address=77.90.31.19:4875 , piece_id=QPTEEI6KQOY3BTIPMZA7RCALZWM7ESRO7ARHJOUK5HOZQ7AUEEMQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.956449663 (394h29m41.727090998s) [noise=true , address=77.90.31.21:4911 , piece_id=SKWN7R5R6DSA4GXMIDGY2IG5EROC737ELJR56BUMGJGJLDLKVNZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.957911109 (394h29m41.728552444s) [noise=true , address=77.90.31.15:4703 , piece_id=FA6NQWQWWHUHNII5I24GAF7CSSYMHCUOVCVN7NOM77OV2NC5I7YQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.958308970 (394h29m41.728950305s) [noise=true , address=77.90.31.13:4642 , piece_id=3NJT6ZKBSQTJCKAVVNMUTGCXTUABCTB7W5BNBU2F6S4IYY4NSQFA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.963606204 (394h29m41.734247539s) [noise=true , address=77.90.31.18:4828 , piece_id=4YRG3357MDWT5H3KB45WD5OVI4HW27AI5SXUZAKO3IATE6MQFMBA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.967122020 (394h29m41.737763365s) [noise=true , address=77.90.31.10:4527 , piece_id=KV3ZMKPLATXPUVYKVY6CD62K5CQYOMPGA7WI3EHIRVXP75ADICFQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:06.968750526 (394h29m41.739391861s) [noise=true , address=77.90.31.12:4619 , piece_id=RLOAYDELUC72UV23GHMEIBNIUTXOZVVDPDTJ2ISQMXJ7ZCZGMR4Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.413127521 (394h29m44.183768856s) [noise=true , address=79.127.219.48:4085 , piece_id=OTON2L4UEES3UQZEKY2GTROOXHPCL4DWQTUYBUAAEKNJSQ4BVVHA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:09.552338803 (394h29m44.322980128s) [noise=true , address=79.127.219.47:4047 , piece_id=RTDQ52WKJEGH2CY2R5ALQXHZTKROI375MLNPUPU2YTSBEDJYONOQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:11.406061841 (394h29m46.176703187s) [noise=true , address=45.91.116.12:4247 , piece_id=BCQZTNCDOVRKM7NDVJ2F2BOQ2UYIRCO52MYUXQ6BO2SEZERWWTDA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:15.829545267 (394h29m50.600186622s) [noise=true , address=79.127.201.215:4347 , piece_id=OIPNODTJOLAOHPNV7DDJ6DDUZXPFWM3JIS7MEU2LEA3QDTBE5RUQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:17.407206378 (394h29m52.177847713s) [noise=true , address=104.36.82.7:4927 , piece_id=BUPIYVM3D4LVEDQBVS76FTSFV3YU72A4JCWI7Y3JKPW35WIZ3F2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:40:27.243985062 (394h30m2.014626397s) [noise=true , address=104.36.82.4:4805 , piece_id=DLFM7RN6SXN3V3XHSS2SFD4FXXAVJ26R76JNPG5XCXNEQDPX2Y2Q, seg_pos={0 0}, upload=315633]: Done uploading piece. err: <nil> Jul 9 19:41:03.876386888 (394h30m38.647028223s) [noise=true , address=104.36.82.35:4615 , piece_id=IIFN264CVUSKUXEUVTWFNLU3E5KS3DPEL5XLLZXJSZBMUHKKX2RA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: expected piece hash; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; write tcp 188.166.225.170:39628->104.36.82.35:4615: use of closed network connection; EOF Jul 9 19:41:03.876621056 (394h30m38.647262421s) [noise=true , address=104.36.82.23:4469 , piece_id=UYQRYJMGIIURUTDBPKO73DWMNGPF3TZ5VAIGIIKJA3KWN4PUXFZA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876686712 (394h30m38.647328047s) [noise=true , address=104.36.82.36:4621 , piece_id=EEDP5JHYYOEESKJWYMMVOPNBSZ52AI2IKLBPALPD727BFBDIOIAA, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:12VY2FEQWX4WPXwgMVBKoANFgqFd1npRBUG8pfQKWVB6gJw96AS, address:104.36.82.36:4621): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 19:41:03.876788527 (394h30m38.647429862s) [noise=true , address=104.36.82.49:4986 , piece_id=KMHGSKASFR7AZCFUML6PTA56H2GPQAPBM2KIGZUFJC5TIQW2S2WQ, seg_pos={0 0}, upload=315633]: Done uploading piece. err: ecclient: upload failed (node:1xiyNnuK7RcA3pYJRz2QStzUVfaU6gVUovxX2gV86xAQ2K9np4, address:104.36.82.49:4986): protocol: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51200->104.36.82.49:4986: use of closed network connection Jul 9 19:41:03.876802763 (394h30m38.647444098s) [seg_pos={0 0}, upload=315633]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 19:41:04.107017769 (394h30m38.877659124s) [upload=315633]: Done uploading segments... Jul 9 19:41:04.107047426 (394h30m38.877688761s) [upload=315633]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 48.458421022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 50.555874129s FAIL (2025-07-09 19:48:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.555895048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X........X.......................X....] duration: 6.250190556s FAIL (2025-07-09 19:48:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.250203822s edge gateway us-select-1

1
...
6321
6322
6323
6324
6325
...
6633