Orbiter History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.XXX..................] duration: 689.82411ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:54:24 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 689.877122ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [..................................XX..XXX...................] duration: 2.264344086s FAIL (2025-08-16 14:54:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.264365386s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................X........XXX..XXXXXX.................] duration: 6.328533145s FAIL (2025-08-16 14:54:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.328557632s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 738.21945ms FAIL (2025-08-16 14:54:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 738.284895ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........X............................X....X........X.....X.] duration: 4.821534s FAIL (2025-08-16 14:53:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 14:53:02.753779620 (189h26m57.184859781s) [upload=197106]: Starting upload Aug 16 14:53:02.753896433 (189h26m57.184976564s) [upload=197106]: Done starting upload Aug 16 14:53:02.754004199 (189h26m57.18508433s) [upload=197106]: Uploading segments... Aug 16 14:53:02.754069675 (189h26m57.185149806s) [upload=197106]: Got next segment. Inline: true Aug 16 14:53:02.754075095 (189h26m57.185155226s) [upload=197106]: Waiting for error group managing segments... Aug 16 14:53:02.754123679 (189h26m57.18520381s) [upload=197106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:53:02.754128227 (189h26m57.185208358s) [upload=197106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:53:02.754130862 (189h26m57.185210993s) [upload=197106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:53:02.970993332 (189h26m57.402073463s) [upload=197106]: Done uploading segments... Aug 16 14:53:02.971011477 (189h26m57.402091608s) [upload=197106]: Upload finished. err: metaclient: Too Many Requests Duration:* 4.821559429s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X.X.XXX..................] duration: 9.541287973s FAIL (2025-08-16 14:53:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 9.541316027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X................................X.X.XXX...X.......X...] duration: 10.344699449s FAIL (2025-08-16 14:53:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after d0d7909526146738ec9341b4dc5a2f5e04cc6a0e4dfa6da2d5efd06d2f0ed0d8 != before d10618a9fa178b81ced5f1ca7db57af99bf9f34e1b256ff968681f875f8e7985 ======LOGS====== Aug 16 14:50:49.763513094 (189h24m44.194593225s) [upload=197065]: Starting upload Aug 16 14:50:49.763605380 (189h24m44.194685511s) [upload=197065]: Done starting upload Aug 16 14:50:49.763717755 (189h24m44.194797936s) [upload=197065]: Uploading segments... Aug 16 14:50:49.764050974 (189h24m44.195131115s) [upload=197065]: Got next segment. Inline: false Aug 16 14:50:49.764187164 (189h24m44.195267345s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:50:49.764197423 (189h24m44.195277585s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:50:51.364767553 (189h24m45.795847684s) [seg_pos={0 0}, upload=197065]: Begin upload segment... Aug 16 14:50:51.364817599 (189h24m45.79589773s) [seg_pos={0 0}, upload=197065]: Done begin upload segment. Aug 16 14:50:51.364841806 (189h24m45.795921937s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.364977305 (189h24m45.796057456s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365290153 (189h24m45.796370284s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365487250 (189h24m45.796567381s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365697864 (189h24m45.796777995s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365922673 (189h24m45.797002805s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366224080 (189h24m45.797304211s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366368466 (189h24m45.797448597s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366596102 (189h24m45.797676224s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366685784 (189h24m45.797765915s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366932015 (189h24m45.798012146s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367158528 (189h24m45.798238659s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367274329 (189h24m45.798354461s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367483991 (189h24m45.798564112s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367581027 (189h24m45.798661158s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367777743 (189h24m45.798857874s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367900227 (189h24m45.798980358s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.368240349 (189h24m45.79932048s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:52.495789086 (189h24m46.926869217s) [upload=197065]: Next returned nil segment Aug 16 14:50:52.495806640 (189h24m46.926886761s) [upload=197065]: Waiting for error group managing segments... Aug 16 14:50:52.779408861 (189h24m47.210488982s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:52.823088436 (189h24m47.254168567s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.339600233 (189h24m47.770680364s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.490333384 (189h24m47.921413515s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.196547946 (189h24m48.627628077s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.415943445 (189h24m48.847023566s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.609113834 (189h24m49.040193965s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.027627813 (189h24m49.458707944s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.341495503 (189h24m49.772575634s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.402311798 (189h24m49.833391929s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.652125648 (189h24m50.083205779s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.279438481 (189h24m50.710518612s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.619303655 (189h24m51.050383786s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921594244 (189h24m51.352674375s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921637477 (189h24m51.352717628s) [seg_pos={0 0}, upload=197065]: Segment reached optimal threshold of 14 pieces. Aug 16 14:50:56.921971766 (189h24m51.353051897s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection Aug 16 14:50:56.921996453 (189h24m51.353076585s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922075054 (189h24m51.353155175s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922178532 (189h24m51.353258663s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: read tcp 188.166.225.170:49128->104.36.82.8:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:50:56.922209171 (189h24m51.353289302s) [seg_pos={0 0}, upload=197065]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:50:56.922304453 (189h24m51.353384584s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:50:56.922311055 (189h24m51.353391186s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:50:57.816555440 (189h24m52.247635591s) [upload=197065]: Done uploading segments... Aug 16 14:50:57.816602470 (189h24m52.247682611s) [upload=197065]: Upload finished. err: <nil> Duration: 10.344726523s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................XXX..XXXXX.................] duration: 6.717001883s FAIL (2025-08-16 14:53:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.717039144s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 480.860383ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:53:27 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 480.898597ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.XXX.................] duration: 749.182405ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:53:27 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 749.209707ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................XX..XXX..................] duration: 2.260980189s FAIL (2025-08-16 14:53:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.261001771s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................X........XXX..XXXXXX................] duration: 6.082669091s FAIL (2025-08-16 14:53:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.082692365s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........X............................X....X........X.....X] duration: 872.849157ms FAIL (2025-08-16 14:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 14:53:02.753779620 (189h26m57.184859781s) [upload=197106]: Starting upload Aug 16 14:53:02.753896433 (189h26m57.184976564s) [upload=197106]: Done starting upload Aug 16 14:53:02.754004199 (189h26m57.18508433s) [upload=197106]: Uploading segments... Aug 16 14:53:02.754069675 (189h26m57.185149806s) [upload=197106]: Got next segment. Inline: true Aug 16 14:53:02.754075095 (189h26m57.185155226s) [upload=197106]: Waiting for error group managing segments... Aug 16 14:53:02.754123679 (189h26m57.18520381s) [upload=197106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:53:02.754128227 (189h26m57.185208358s) [upload=197106]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:53:02.754130862 (189h26m57.185210993s) [upload=197106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:53:02.970993332 (189h26m57.402073463s) [upload=197106]: Done uploading segments... Aug 16 14:53:02.971011477 (189h26m57.402091608s) [upload=197106]: Upload finished. err: metaclient: Too Many Requests Duration:* 872.873163ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X................................X.X.XXX...X.......X..] duration: 10.296409939s FAIL (2025-08-16 14:52:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after d0d7909526146738ec9341b4dc5a2f5e04cc6a0e4dfa6da2d5efd06d2f0ed0d8 != before d10618a9fa178b81ced5f1ca7db57af99bf9f34e1b256ff968681f875f8e7985 ======LOGS====== Aug 16 14:50:49.763513094 (189h24m44.194593225s) [upload=197065]: Starting upload Aug 16 14:50:49.763605380 (189h24m44.194685511s) [upload=197065]: Done starting upload Aug 16 14:50:49.763717755 (189h24m44.194797936s) [upload=197065]: Uploading segments... Aug 16 14:50:49.764050974 (189h24m44.195131115s) [upload=197065]: Got next segment. Inline: false Aug 16 14:50:49.764187164 (189h24m44.195267345s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:50:49.764197423 (189h24m44.195277585s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:50:51.364767553 (189h24m45.795847684s) [seg_pos={0 0}, upload=197065]: Begin upload segment... Aug 16 14:50:51.364817599 (189h24m45.79589773s) [seg_pos={0 0}, upload=197065]: Done begin upload segment. Aug 16 14:50:51.364841806 (189h24m45.795921937s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.364977305 (189h24m45.796057456s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365290153 (189h24m45.796370284s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365487250 (189h24m45.796567381s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365697864 (189h24m45.796777995s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365922673 (189h24m45.797002805s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366224080 (189h24m45.797304211s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366368466 (189h24m45.797448597s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366596102 (189h24m45.797676224s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366685784 (189h24m45.797765915s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366932015 (189h24m45.798012146s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367158528 (189h24m45.798238659s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367274329 (189h24m45.798354461s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367483991 (189h24m45.798564112s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367581027 (189h24m45.798661158s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367777743 (189h24m45.798857874s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367900227 (189h24m45.798980358s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.368240349 (189h24m45.79932048s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:52.495789086 (189h24m46.926869217s) [upload=197065]: Next returned nil segment Aug 16 14:50:52.495806640 (189h24m46.926886761s) [upload=197065]: Waiting for error group managing segments... Aug 16 14:50:52.779408861 (189h24m47.210488982s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:52.823088436 (189h24m47.254168567s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.339600233 (189h24m47.770680364s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.490333384 (189h24m47.921413515s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.196547946 (189h24m48.627628077s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.415943445 (189h24m48.847023566s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.609113834 (189h24m49.040193965s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.027627813 (189h24m49.458707944s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.341495503 (189h24m49.772575634s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.402311798 (189h24m49.833391929s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.652125648 (189h24m50.083205779s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.279438481 (189h24m50.710518612s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.619303655 (189h24m51.050383786s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921594244 (189h24m51.352674375s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921637477 (189h24m51.352717628s) [seg_pos={0 0}, upload=197065]: Segment reached optimal threshold of 14 pieces. Aug 16 14:50:56.921971766 (189h24m51.353051897s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection Aug 16 14:50:56.921996453 (189h24m51.353076585s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922075054 (189h24m51.353155175s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922178532 (189h24m51.353258663s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: read tcp 188.166.225.170:49128->104.36.82.8:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:50:56.922209171 (189h24m51.353289302s) [seg_pos={0 0}, upload=197065]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:50:56.922304453 (189h24m51.353384584s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:50:56.922311055 (189h24m51.353391186s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:50:57.816555440 (189h24m52.247635591s) [upload=197065]: Done uploading segments... Aug 16 14:50:57.816602470 (189h24m52.247682611s) [upload=197065]: Upload finished. err: <nil> Duration: 10.296433084s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 762.180677ms FAIL (2025-08-16 14:52:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 762.20826ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X.X.XXX.................] duration: 7.214231339s FAIL (2025-08-16 14:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.214257009s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.XXX................] duration: 708.961718ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:52:50 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 709.001925ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 498.627337ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:52:30 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 498.655842ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................XXX..XXXXX................] duration: 7.210105722s FAIL (2025-08-16 14:52:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.210131862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........X............................X....X........X.....] duration: 2.515495444s FAIL (2025-08-16 14:52:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 14:48:50.108818187 (189h22m44.539898318s) [upload=197031]: Starting upload Aug 16 14:48:50.108864806 (189h22m44.539944937s) [upload=197031]: Done starting upload Aug 16 14:48:50.108917006 (189h22m44.539997137s) [upload=197031]: Uploading segments... Aug 16 14:48:50.108954588 (189h22m44.540034719s) [upload=197031]: Got next segment. Inline: true Aug 16 14:48:50.108958365 (189h22m44.540038496s) [upload=197031]: Waiting for error group managing segments... Aug 16 14:48:50.108989224 (189h22m44.540069355s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:48:50.108993232 (189h22m44.540073353s) [upload=197031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:48:50.108995646 (189h22m44.540075777s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:48:54.461533727 (189h22m48.892613858s) [upload=197031]: Done uploading segments... Aug 16 14:48:54.461551360 (189h22m48.892631481s) [upload=197031]: Upload finished. err: <nil> Duration:* 2.515529541s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X.X.XXX................] duration: 8.560990404s FAIL (2025-08-16 14:52:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 8.561016364s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.XXX...............] duration: 724.492775ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:52:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 724.518434ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................XX..XXX.................] duration: 2.277110601s FAIL (2025-08-16 14:52:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.277130941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................X........XXX..XXXXXX...............] duration: 6.232584033s FAIL (2025-08-16 14:52:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.232637836s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X................................X.X.XXX...X.......X.] duration: 14.517305764s FAIL (2025-08-16 14:52:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after d0d7909526146738ec9341b4dc5a2f5e04cc6a0e4dfa6da2d5efd06d2f0ed0d8 != before d10618a9fa178b81ced5f1ca7db57af99bf9f34e1b256ff968681f875f8e7985 ======LOGS====== Aug 16 14:50:49.763513094 (189h24m44.194593225s) [upload=197065]: Starting upload Aug 16 14:50:49.763605380 (189h24m44.194685511s) [upload=197065]: Done starting upload Aug 16 14:50:49.763717755 (189h24m44.194797936s) [upload=197065]: Uploading segments... Aug 16 14:50:49.764050974 (189h24m44.195131115s) [upload=197065]: Got next segment. Inline: false Aug 16 14:50:49.764187164 (189h24m44.195267345s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:50:49.764197423 (189h24m44.195277585s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:50:51.364767553 (189h24m45.795847684s) [seg_pos={0 0}, upload=197065]: Begin upload segment... Aug 16 14:50:51.364817599 (189h24m45.79589773s) [seg_pos={0 0}, upload=197065]: Done begin upload segment. Aug 16 14:50:51.364841806 (189h24m45.795921937s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.364977305 (189h24m45.796057456s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365290153 (189h24m45.796370284s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365487250 (189h24m45.796567381s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365697864 (189h24m45.796777995s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365922673 (189h24m45.797002805s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366224080 (189h24m45.797304211s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366368466 (189h24m45.797448597s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366596102 (189h24m45.797676224s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366685784 (189h24m45.797765915s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366932015 (189h24m45.798012146s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367158528 (189h24m45.798238659s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367274329 (189h24m45.798354461s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367483991 (189h24m45.798564112s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367581027 (189h24m45.798661158s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367777743 (189h24m45.798857874s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367900227 (189h24m45.798980358s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.368240349 (189h24m45.79932048s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:52.495789086 (189h24m46.926869217s) [upload=197065]: Next returned nil segment Aug 16 14:50:52.495806640 (189h24m46.926886761s) [upload=197065]: Waiting for error group managing segments... Aug 16 14:50:52.779408861 (189h24m47.210488982s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:52.823088436 (189h24m47.254168567s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.339600233 (189h24m47.770680364s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.490333384 (189h24m47.921413515s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.196547946 (189h24m48.627628077s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.415943445 (189h24m48.847023566s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.609113834 (189h24m49.040193965s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.027627813 (189h24m49.458707944s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.341495503 (189h24m49.772575634s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.402311798 (189h24m49.833391929s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.652125648 (189h24m50.083205779s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.279438481 (189h24m50.710518612s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.619303655 (189h24m51.050383786s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921594244 (189h24m51.352674375s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921637477 (189h24m51.352717628s) [seg_pos={0 0}, upload=197065]: Segment reached optimal threshold of 14 pieces. Aug 16 14:50:56.921971766 (189h24m51.353051897s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection Aug 16 14:50:56.921996453 (189h24m51.353076585s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922075054 (189h24m51.353155175s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922178532 (189h24m51.353258663s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: read tcp 188.166.225.170:49128->104.36.82.8:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:50:56.922209171 (189h24m51.353289302s) [seg_pos={0 0}, upload=197065]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:50:56.922304453 (189h24m51.353384584s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:50:56.922311055 (189h24m51.353391186s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:50:57.816555440 (189h24m52.247635591s) [upload=197065]: Done uploading segments... Aug 16 14:50:57.816602470 (189h24m52.247682611s) [upload=197065]: Upload finished. err: <nil> Duration: 14.517332554s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.XXX..............] duration: 706.76209ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:51:54 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 706.827735ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................................XXX..XXXXX...............] duration: 6.887617921s FAIL (2025-08-16 14:51:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.887645043s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XX..XXX................] duration: 2.423615635s FAIL (2025-08-16 14:51:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.423638999s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X.X.XXX...............] duration: 7.63137234s FAIL (2025-08-16 14:51:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.631400934s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 777.758894ms FAIL (2025-08-16 14:51:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 777.778341ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [............X............................X....X........X....] duration: 7.211447169s FAIL (2025-08-16 14:51:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 14:48:50.108818187 (189h22m44.539898318s) [upload=197031]: Starting upload Aug 16 14:48:50.108864806 (189h22m44.539944937s) [upload=197031]: Done starting upload Aug 16 14:48:50.108917006 (189h22m44.539997137s) [upload=197031]: Uploading segments... Aug 16 14:48:50.108954588 (189h22m44.540034719s) [upload=197031]: Got next segment. Inline: true Aug 16 14:48:50.108958365 (189h22m44.540038496s) [upload=197031]: Waiting for error group managing segments... Aug 16 14:48:50.108989224 (189h22m44.540069355s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:48:50.108993232 (189h22m44.540073353s) [upload=197031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:48:50.108995646 (189h22m44.540075777s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:48:54.461533727 (189h22m48.892613858s) [upload=197031]: Done uploading segments... Aug 16 14:48:54.461551360 (189h22m48.892631481s) [upload=197031]: Upload finished. err: <nil> Duration:* 7.211501924s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 712.530498ms FAIL (2025-08-16 14:51:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 712.564404ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................XX..XXX...............] duration: 2.346935645s FAIL (2025-08-16 14:51:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.346960462s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 479.314112ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:51:18 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 479.341635ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................X........XXX..XXXXXX..............] duration: 6.221726994s FAIL (2025-08-16 14:51:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.221749307s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................XXX..XXXXX..............] duration: 7.13178239s FAIL (2025-08-16 14:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.13181896s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XX..XXX..............] duration: 2.461551741s FAIL (2025-08-16 14:51:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.461575376s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X................................X.X.XXX...X.......X] duration: 11.182481324s FAIL (2025-08-16 14:51:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after d0d7909526146738ec9341b4dc5a2f5e04cc6a0e4dfa6da2d5efd06d2f0ed0d8 != before d10618a9fa178b81ced5f1ca7db57af99bf9f34e1b256ff968681f875f8e7985 ======LOGS====== Aug 16 14:50:49.763513094 (189h24m44.194593225s) [upload=197065]: Starting upload Aug 16 14:50:49.763605380 (189h24m44.194685511s) [upload=197065]: Done starting upload Aug 16 14:50:49.763717755 (189h24m44.194797936s) [upload=197065]: Uploading segments... Aug 16 14:50:49.764050974 (189h24m44.195131115s) [upload=197065]: Got next segment. Inline: false Aug 16 14:50:49.764187164 (189h24m44.195267345s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:50:49.764197423 (189h24m44.195277585s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:50:51.364767553 (189h24m45.795847684s) [seg_pos={0 0}, upload=197065]: Begin upload segment... Aug 16 14:50:51.364817599 (189h24m45.79589773s) [seg_pos={0 0}, upload=197065]: Done begin upload segment. Aug 16 14:50:51.364841806 (189h24m45.795921937s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.364977305 (189h24m45.796057456s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365290153 (189h24m45.796370284s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365487250 (189h24m45.796567381s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365697864 (189h24m45.796777995s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.365922673 (189h24m45.797002805s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366224080 (189h24m45.797304211s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366368466 (189h24m45.797448597s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366596102 (189h24m45.797676224s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366685784 (189h24m45.797765915s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.366932015 (189h24m45.798012146s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367158528 (189h24m45.798238659s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367274329 (189h24m45.798354461s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367483991 (189h24m45.798564112s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367581027 (189h24m45.798661158s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367777743 (189h24m45.798857874s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.367900227 (189h24m45.798980358s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:51.368240349 (189h24m45.79932048s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Uploading piece... Aug 16 14:50:52.495789086 (189h24m46.926869217s) [upload=197065]: Next returned nil segment Aug 16 14:50:52.495806640 (189h24m46.926886761s) [upload=197065]: Waiting for error group managing segments... Aug 16 14:50:52.779408861 (189h24m47.210488982s) [noise=true , address=79.127.205.233:4627 , piece_id=5MWB7AGMAZJH5FMNXC3P3X4DV2AHGYZLOTAFEYRO6VZAPE6JPYCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:52.823088436 (189h24m47.254168567s) [noise=true , address=66.23.193.175:4561 , piece_id=6MT3UYL66TJERLV735MX5SE67VP7WCJYPPUCCPWMV4KSGIJPM64A, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.339600233 (189h24m47.770680364s) [noise=true , address=156.146.43.225:4457 , piece_id=AVLJABYHWDQ433A4HNTOOCEBQWKKFFP3USMZDFLD5YKOHMNU4SVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:53.490333384 (189h24m47.921413515s) [noise=true , address=104.36.82.44:4884 , piece_id=NQUYB6MIXEBCCRWPTUKFWNS654UL7S7IIWNZK3ZYOI7YLI56WHOA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.196547946 (189h24m48.627628077s) [noise=true , address=77.90.31.14:4664 , piece_id=IFQMJYAY5U4OXKVZBFQ6FZA4N5CKHTNNUE5DU56TILJFBR5MVSWQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.415943445 (189h24m48.847023566s) [noise=true , address=77.90.31.11:4567 , piece_id=NS2BSBDE6MZMHEOY2EWTM53GH27T6CS7NIPSTRX72G47XKUIHYNQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:54.609113834 (189h24m49.040193965s) [noise=true , address=77.90.31.15:4700 , piece_id=XZATFU6FMY4QTZ2OF7SLNVM3BN2VEDSVWPWXNSFSEF6PLNF7BJIQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.027627813 (189h24m49.458707944s) [noise=true , address=45.91.116.14:4204 , piece_id=2KFTPCCGZANQTCXFCB3JI7DL34HMN3DLEQEATG4XZ2MR5DVR3BUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.341495503 (189h24m49.772575634s) [noise=true , address=104.36.82.5:4849 , piece_id=RQI6I3ALHB3V5EBN5ZVZ7N3SY2FEPYAU7WYIJBVG5Z7X65QMYDSQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.402311798 (189h24m49.833391929s) [noise=true , address=104.36.82.35:4584 , piece_id=JTPQKVSEYNPELYX6HGWXAQEVNMPGVUYJMBXI2EMYOKVE4WW25XCA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:55.652125648 (189h24m50.083205779s) [noise=true , address=77.90.31.19:4857 , piece_id=M3PHCDDA77XUDVR52KGAPETCN2NYXN4KZ7OTDWWD7OY4MUQ7BJUA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.279438481 (189h24m50.710518612s) [noise=true , address=77.90.31.16:4747 , piece_id=QAGWFSLADPLDAZN2BMKNGGOZN6VKGZQOIUZIFD76VFORB6EIJYVA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.619303655 (189h24m51.050383786s) [noise=true , address=104.36.82.60:4363 , piece_id=HVFQEPCMXJZDE57MZOSHHDUUPNLVJNCS3B7WIKCIPKWJGHONA2TQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921594244 (189h24m51.352674375s) [noise=true , address=77.90.31.18:4838 , piece_id=GUYV54UTB4KW3DWGKDYSXQT4ZR4NIX6YDL4GVAI4XU2VJS7RDDUQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: <nil> Aug 16 14:50:56.921637477 (189h24m51.352717628s) [seg_pos={0 0}, upload=197065]: Segment reached optimal threshold of 14 pieces. Aug 16 14:50:56.921971766 (189h24m51.353051897s) [noise=true , address=77.90.31.21:4882 , piece_id=XWOKKLWFUKRVWU2HMSYJR7P2ATAY6EB4YFAY2Y6EN45ONOJE4XZQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44084->77.90.31.21:4882: use of closed network connection Aug 16 14:50:56.921996453 (189h24m51.353076585s) [noise=true , address=104.36.82.34:4576 , piece_id=HBJ6CY63XED5OSZM56EVAFOHTJTH3GF7RJ2AOJ4OUID7DZ7DLYQQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122bMTJLxntv7VEY56zSPLZm5wZHAH3rnE8UZHNCjRhd7P9oKus): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922075054 (189h24m51.353155175s) [noise=true , address=104.36.82.23:4453 , piece_id=MTEZCCM433NYTN345OYWD2AQ5BYO32HMBXRKSTWBUVRREVXS5OTQ, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 16 14:50:56.922178532 (189h24m51.353258663s) [noise=true , address=104.36.82.8:4974 , piece_id=PEZRM4XDIZMAS5JXVVSBHG4HWYV4IXGIBY5T5FNM53T5V2SDXHSA, seg_pos={0 0}, upload=197065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: read tcp 188.166.225.170:49128->104.36.82.8:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:50:56.922209171 (189h24m51.353289302s) [seg_pos={0 0}, upload=197065]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:50:56.922304453 (189h24m51.353384584s) [upload=197065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:50:56.922311055 (189h24m51.353391186s) [upload=197065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:50:57.816555440 (189h24m52.247635591s) [upload=197065]: Done uploading segments... Aug 16 14:50:57.816602470 (189h24m52.247682611s) [upload=197065]: Upload finished. err: <nil> Duration: 11.182511472s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............X............................X....X........X...] duration: 10.005940029s FAIL (2025-08-16 14:50:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 14:48:50.108818187 (189h22m44.539898318s) [upload=197031]: Starting upload Aug 16 14:48:50.108864806 (189h22m44.539944937s) [upload=197031]: Done starting upload Aug 16 14:48:50.108917006 (189h22m44.539997137s) [upload=197031]: Uploading segments... Aug 16 14:48:50.108954588 (189h22m44.540034719s) [upload=197031]: Got next segment. Inline: true Aug 16 14:48:50.108958365 (189h22m44.540038496s) [upload=197031]: Waiting for error group managing segments... Aug 16 14:48:50.108989224 (189h22m44.540069355s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:48:50.108993232 (189h22m44.540073353s) [upload=197031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:48:50.108995646 (189h22m44.540075777s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:48:54.461533727 (189h22m48.892613858s) [upload=197031]: Done uploading segments... Aug 16 14:48:54.461551360 (189h22m48.892631481s) [upload=197031]: Upload finished. err: <nil> Duration:* 10.005963583s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.XXX.............] duration: 780.533743ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:50:53 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 780.559773ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 603.871894ms remote: "tcp:3.109.243.18:443" response headers: Date=Sat, 16 Aug 2025 14:40:59 GMT Content-Type=text/html Content-Length=164 X-Cluster-Name=ap-south-1-prod-hosting-red target: https://www.storj.io/dcs-satellites FAIL (2025-08-16 14:50:50 UTC) orbiter warning: unexpected status code "504 Gateway Time-out" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Duration: 603.897855ms core website
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 766.245883ms FAIL (2025-08-16 14:50:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 766.277305ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X.X.XXX..............] duration: 8.541388567s FAIL (2025-08-16 14:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 8.541448221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........................X........XXX..XXXXXX.............] duration: 6.321832327s FAIL (2025-08-16 14:50:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.321874158s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [........................................XX..XXX.............] duration: 2.388257621s FAIL (2025-08-16 14:50:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.388279752s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 960.939188ms remote: "tcp:3.109.243.18:443" response headers: Date=Sat, 16 Aug 2025 14:40:59 GMT Content-Type=text/html Content-Length=164 X-Cluster-Name=ap-south-1-prod-hosting-red target: https://www.storj.io/dcs-satellites FAIL (2025-08-16 14:50:30 UTC) orbiter warning: unexpected status code "504 Gateway Time-out" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Duration: 960.959538ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................................XXX..XXXXX.............] duration: 7.726580562s FAIL (2025-08-16 14:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.726608264s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 511.811542ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:50:06 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 511.835388ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X................................X.X.XXX...X.......] duration: 17.669364288s FAIL (2025-08-16 14:50:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after c3eada3f50ffa3c11513e2f0166ad335d65133e45301768bdaeafbdeb98e0868 != before 529376e75b4b13b4082758ef0931f0e6e7b681d97e7cde9f1fe7f7e69cf6fa70 ======LOGS====== Aug 16 14:43:55.670363191 (189h17m50.101443352s) [upload=196939]: Starting upload Aug 16 14:43:55.670452171 (189h17m50.101532302s) [upload=196939]: Done starting upload Aug 16 14:43:55.670540660 (189h17m50.101620831s) [upload=196939]: Uploading segments... Aug 16 14:43:55.670879940 (189h17m50.101960071s) [upload=196939]: Got next segment. Inline: false Aug 16 14:43:55.670925507 (189h17m50.102005648s) [upload=196939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:43:55.670932230 (189h17m50.102012391s) [upload=196939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:43:56.443656294 (189h17m50.874736425s) [seg_pos={0 0}, upload=196939]: Begin upload segment... Aug 16 14:43:56.443726178 (189h17m50.874806309s) [seg_pos={0 0}, upload=196939]: Done begin upload segment. Aug 16 14:43:56.443757087 (189h17m50.874837218s) [noise=true , address=104.36.82.58:4292 , piece_id=NPC2CT7RSFNZUD5IXMJVBYPSB73J2ILYWNQ3XQJ3Q4JTEUFETOHA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.443818485 (189h17m50.874898616s) [noise=true , address=104.36.82.23:4455 , piece_id=PSXR7K4QRLJRZESS2RWOAQIKONU3VY5ETFWHGANE6W6XG4LSYYAA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.444335153 (189h17m50.875415294s) [noise=true , address=104.36.82.2:4744 , piece_id=GYGAJDJEBB2UDAQQ6XUKXYUWFPMIRVU3OVS6SP2MONKRTIM7QCSA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.444537851 (189h17m50.875617972s) [noise=true , address=77.90.31.21:4906 , piece_id=J2IBKKABRRT2D76GQUGV6HTGZ7CFGNSGENBIPKRVHRRPNAOA4JLA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.444765065 (189h17m50.875845216s) [noise=true , address=66.23.193.175:4558 , piece_id=KBQUCSKIM7VTFX2YKTU25U7ZX7H735CWFJYBBVXOAONQDJXP5WSA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.445035132 (189h17m50.876115263s) [noise=true , address=45.91.116.12:4232 , piece_id=JKONF6EERFNEAN7IQ774NLMAIXQLIR3ULOPS5MX4LJP6ENKBSBEQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.445158549 (189h17m50.87623867s) [noise=true , address=104.36.82.34:4562 , piece_id=SBGHSUWZ5BMHPPINYVZUMUKLIEAOIXLE7FDAMOBN43Z3LDEZYWBA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.445529368 (189h17m50.876609499s) [noise=true , address=79.127.219.38:4035 , piece_id=COBSVV4LKOK2YTN2F46X6SEES4GQIATNSJ7T7DORA2TSAIFXV67A, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.445538917 (189h17m50.876619048s) [noise=true , address=79.127.220.101:4073 , piece_id=HKAOL5FEY4PUYTP2KKSRYW62SEHPE2IWS5IV23KA3QCOVK2MMZJQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.446005199 (189h17m50.87708533s) [noise=true , address=77.90.31.13:4632 , piece_id=EV2VSYUA75IDBLCTMRYE3LBUGBIC3HTFR3BJGJZTROZ6NYNDTJVQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.446043291 (189h17m50.877123422s) [noise=true , address=45.91.116.11:4135 , piece_id=75FSDT2DW3WUHXNT7327F6IUY4YNHXORTALFRWUX6L2BXW4NIOOQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.446472182 (189h17m50.877552313s) [noise=true , address=45.91.116.10:4107 , piece_id=PIQWBCB2XOC4YTNGA3ML4RWIRIGLILOZB5GYQTWEZLW7TFV2KENQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.446663919 (189h17m50.87774405s) [noise=true , address=104.36.82.36:4619 , piece_id=2EAIFF5K43YA5YW6IKQ3CM7LF5MVE7FLY247U6B5H622PS72TYZA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.446851689 (189h17m50.87793182s) [noise=true , address=104.36.82.60:4360 , piece_id=MB4V57IGWRB3KB7CJVHJP52GD7BK7TTS5HLGFYVKHN6SJIANBENQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.447094192 (189h17m50.878174323s) [noise=true , address=104.36.82.5:4863 , piece_id=6AQ2I3VGTJTN2XGPUJU7CWSGE3FOMR7XCEHHYK5I35OI3JM6RZEA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.447256262 (189h17m50.878336393s) [noise=true , address=104.36.82.38:4691 , piece_id=QMBT7DWMFXPW7MIUNNQ2357ELM7CVILWJIO5JSUN6PC65GGOJ6HQ, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.447521770 (189h17m50.878601901s) [noise=true , address=104.36.82.44:4893 , piece_id=RRYJSUN62IXTC452WKCCFYNG6GTGESKMGCGGZUASRNZ3QB5FA2RA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:56.447932055 (189h17m50.879012187s) [noise=true , address=104.36.82.59:4327 , piece_id=XUR4D7WBQR7YDHHCPBTBO5MMQ475WE2P6EUGQ4D4K2K6DZPZY2KA, seg_pos={0 0}, upload=196939]: Uploading piece... Aug 16 14:43:57.614344119 (189h17m52.04542425s) [upload=196939]: Next returned nil segment Aug 16 14:43:57.614362283 (189h17m52.045442414s) [upload=196939]: Waiting for error group managing segments... Aug 16 14:43:57.868927165 (189h17m52.300007286s) [noise=true , address=66.23.193.175:4558 , piece_id=KBQUCSKIM7VTFX2YKTU25U7ZX7H735CWFJYBBVXOAONQDJXP5WSA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.371065292 (189h17m52.802145413s) [noise=true , address=104.36.82.2:4744 , piece_id=GYGAJDJEBB2UDAQQ6XUKXYUWFPMIRVU3OVS6SP2MONKRTIM7QCSA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.427518319 (189h17m52.85859845s) [noise=true , address=79.127.219.38:4035 , piece_id=COBSVV4LKOK2YTN2F46X6SEES4GQIATNSJ7T7DORA2TSAIFXV67A, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.494262908 (189h17m52.925343039s) [noise=true , address=104.36.82.36:4619 , piece_id=2EAIFF5K43YA5YW6IKQ3CM7LF5MVE7FLY247U6B5H622PS72TYZA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.522936932 (189h17m52.954017063s) [noise=true , address=104.36.82.5:4863 , piece_id=6AQ2I3VGTJTN2XGPUJU7CWSGE3FOMR7XCEHHYK5I35OI3JM6RZEA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.534661104 (189h17m52.965741235s) [noise=true , address=104.36.82.38:4691 , piece_id=QMBT7DWMFXPW7MIUNNQ2357ELM7CVILWJIO5JSUN6PC65GGOJ6HQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.561626490 (189h17m52.992706621s) [noise=true , address=104.36.82.34:4562 , piece_id=SBGHSUWZ5BMHPPINYVZUMUKLIEAOIXLE7FDAMOBN43Z3LDEZYWBA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.950317829 (189h17m53.38139796s) [noise=true , address=77.90.31.21:4906 , piece_id=J2IBKKABRRT2D76GQUGV6HTGZ7CFGNSGENBIPKRVHRRPNAOA4JLA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.966879998 (189h17m53.397960119s) [noise=true , address=104.36.82.58:4292 , piece_id=NPC2CT7RSFNZUD5IXMJVBYPSB73J2ILYWNQ3XQJ3Q4JTEUFETOHA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:58.999308077 (189h17m53.430388208s) [noise=true , address=45.91.116.10:4107 , piece_id=PIQWBCB2XOC4YTNGA3ML4RWIRIGLILOZB5GYQTWEZLW7TFV2KENQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:59.006402410 (189h17m53.437482531s) [noise=true , address=104.36.82.59:4327 , piece_id=XUR4D7WBQR7YDHHCPBTBO5MMQ475WE2P6EUGQ4D4K2K6DZPZY2KA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:59.078098182 (189h17m53.509178313s) [noise=true , address=79.127.220.101:4073 , piece_id=HKAOL5FEY4PUYTP2KKSRYW62SEHPE2IWS5IV23KA3QCOVK2MMZJQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:59.294644691 (189h17m53.725724822s) [noise=true , address=45.91.116.12:4232 , piece_id=JKONF6EERFNEAN7IQ774NLMAIXQLIR3ULOPS5MX4LJP6ENKBSBEQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:59.416247194 (189h17m53.847327325s) [noise=true , address=77.90.31.13:4632 , piece_id=EV2VSYUA75IDBLCTMRYE3LBUGBIC3HTFR3BJGJZTROZ6NYNDTJVQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: <nil> Aug 16 14:43:59.416321306 (189h17m53.847401477s) [seg_pos={0 0}, upload=196939]: Segment reached optimal threshold of 14 pieces. Aug 16 14:43:59.416487965 (189h17m53.847568086s) [noise=true , address=104.36.82.60:4360 , piece_id=MB4V57IGWRB3KB7CJVHJP52GD7BK7TTS5HLGFYVKHN6SJIANBENQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: context canceled; context canceled Aug 16 14:43:59.416569230 (189h17m53.847649361s) [noise=true , address=45.91.116.11:4135 , piece_id=75FSDT2DW3WUHXNT7327F6IUY4YNHXORTALFRWUX6L2BXW4NIOOQ, seg_pos={0 0}, upload=196939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:194doyGN13Ha8hsyjrERgTZqdyzFhdTQyiTGQg9rFA7CxvKmkM): protocol: expected piece hash; context canceled; EOF Aug 16 14:43:59.416603105 (189h17m53.847683236s) [noise=true , address=104.36.82.44:4893 , piece_id=RRYJSUN62IXTC452WKCCFYNG6GTGESKMGCGGZUASRNZ3QB5FA2RA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:43:59.416645757 (189h17m53.847725888s) [noise=true , address=104.36.82.23:4455 , piece_id=PSXR7K4QRLJRZESS2RWOAQIKONU3VY5ETFWHGANE6W6XG4LSYYAA, seg_pos={0 0}, upload=196939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm): protocol: write tcp 188.166.225.170:47650->104.36.82.23:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:43:59.416659022 (189h17m53.847739153s) [seg_pos={0 0}, upload=196939]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 14:43:59.416715060 (189h17m53.847795181s) [upload=196939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 14:43:59.416718606 (189h17m53.847798727s) [upload=196939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:44:00.708882170 (189h17m55.139962302s) [upload=196939]: Done uploading segments... Aug 16 14:44:00.708911446 (189h17m55.139991577s) [upload=196939]: Upload finished. err: <nil> Duration: 17.669390427s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............X............................X....X........X..] duration: 10.701971586s FAIL (2025-08-16 14:50:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 16 14:48:50.108818187 (189h22m44.539898318s) [upload=197031]: Starting upload Aug 16 14:48:50.108864806 (189h22m44.539944937s) [upload=197031]: Done starting upload Aug 16 14:48:50.108917006 (189h22m44.539997137s) [upload=197031]: Uploading segments... Aug 16 14:48:50.108954588 (189h22m44.540034719s) [upload=197031]: Got next segment. Inline: true Aug 16 14:48:50.108958365 (189h22m44.540038496s) [upload=197031]: Waiting for error group managing segments... Aug 16 14:48:50.108989224 (189h22m44.540069355s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:48:50.108993232 (189h22m44.540073353s) [upload=197031]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:48:50.108995646 (189h22m44.540075777s) [upload=197031]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:48:54.461533727 (189h22m48.892613858s) [upload=197031]: Done uploading segments... Aug 16 14:48:54.461551360 (189h22m48.892631481s) [upload=197031]: Upload finished. err: <nil> Duration:* 10.701991724s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)