Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X.XXX....................................................] duration: 10.060833321s FAIL (2025-08-16 15:18:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 10.060860462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.X.XXX...X.......X...........X......X...X.X......X..] duration: 31.697074354s FAIL (2025-08-16 15:18:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: pending object missing ======LOGS====== Aug 16 15:15:40.748806291 (189h49m35.179886422s) [upload=197494]: Starting upload Aug 16 15:15:40.748847630 (189h49m35.179927761s) [upload=197494]: Done starting upload Aug 16 15:15:40.748901363 (189h49m35.179981494s) [upload=197494]: Uploading segments... Aug 16 15:15:40.749037092 (189h49m35.180117223s) [upload=197494]: Got next segment. Inline: false Aug 16 15:15:40.749062481 (189h49m35.180142612s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:15:40.749066799 (189h49m35.18014693s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:15:42.706876320 (189h49m37.137956451s) [seg_pos={0 0}, upload=197494]: Begin upload segment... Aug 16 15:15:42.706920024 (189h49m37.138000145s) [seg_pos={0 0}, upload=197494]: Done begin upload segment. Aug 16 15:15:42.706944010 (189h49m37.138024131s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.706947897 (189h49m37.138028028s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707294150 (189h49m37.138374271s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707375695 (189h49m37.138455826s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707557744 (189h49m37.138637865s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707816520 (189h49m37.138896651s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708070395 (189h49m37.139150526s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708390919 (189h49m37.13947105s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708609336 (189h49m37.139689467s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708677527 (189h49m37.139757658s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708801684 (189h49m37.139881815s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708994624 (189h49m37.140074755s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709189947 (189h49m37.140270078s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709376594 (189h49m37.140456725s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709548043 (189h49m37.140628174s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710026518 (189h49m37.141106659s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710236450 (189h49m37.141316581s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710434188 (189h49m37.141514319s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:43.864312147 (189h49m38.295392278s) [upload=197494]: Next returned nil segment Aug 16 15:15:43.864332587 (189h49m38.295412718s) [upload=197494]: Waiting for error group managing segments... Aug 16 15:15:44.262984190 (189h49m38.694064321s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.720332763 (189h49m39.151412884s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.762855765 (189h49m39.193935886s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.888679494 (189h49m39.319759625s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.933456339 (189h49m39.36453647s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:45.030036655 (189h49m39.461116776s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.218319864 (189h49m42.649399995s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.266402625 (189h49m42.697482756s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.459775572 (189h49m42.890855703s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.757443534 (189h49m43.188523665s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:49.019151742 (189h49m43.450231873s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.176658524 (189h49m45.607738655s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.823351466 (189h49m46.254431597s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022924656 (189h49m46.454004787s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022953922 (189h49m46.454034053s) [seg_pos={0 0}, upload=197494]: Segment reached optimal threshold of 14 pieces. Aug 16 15:15:52.023339270 (189h49m46.454419401s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi): protocol: write tcp 188.166.225.170:38694->104.36.82.39:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023410757 (189h49m46.454490888s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ): protocol: rpc: dial tcp 104.36.82.58:4286: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023463979 (189h49m46.4545441s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tf9N11h9nLjVCMDpqacHE4qT6NYoPAGWLPbkLX14TZPp8dUV9): protocol: rpc: dial tcp 104.36.82.59:4308: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023513142 (189h49m46.454593273s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH): protocol: write tcp 188.166.225.170:50732->104.36.82.38:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023535665 (189h49m46.454615796s) [seg_pos={0 0}, upload=197494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:15:52.023629966 (189h49m46.454710137s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:15:52.023637009 (189h49m46.45471717s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:16:02.791852702 (189h49m57.222932823s) [upload=197494]: Done uploading segments... Aug 16 15:16:02.791875636 (189h49m57.222955757s) [upload=197494]: Upload finished. err: metaclient: pending object missing Duration: 31.697106796s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....X.XXX..................................................] duration: 776.662332ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:18:29 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 776.68217ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX..XXXXXX................................................] duration: 6.572928545s FAIL (2025-08-16 15:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.572950736s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...XX..XXX..................................................] duration: 2.262461103s FAIL (2025-08-16 15:18:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.26247547s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 498.367938ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:18:23 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 498.391734ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 747.805557ms FAIL (2025-08-16 15:18:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 747.832679ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XXX..XXXXX................................................] duration: 7.420669417s FAIL (2025-08-16 15:18:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.420701678s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....XX..XXX.................................................] duration: 2.30461921s FAIL (2025-08-16 15:17:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.304640041s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 787.740326ms FAIL (2025-08-16 15:17:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 787.769884ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......X.XXX.................................................] duration: 725.618005ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:17:39 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 725.640759ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X.X.XXX...X.......X...........X......X...X.X......X.] duration: 39.68902112s FAIL (2025-08-16 15:17:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: pending object missing ======LOGS====== Aug 16 15:15:40.748806291 (189h49m35.179886422s) [upload=197494]: Starting upload Aug 16 15:15:40.748847630 (189h49m35.179927761s) [upload=197494]: Done starting upload Aug 16 15:15:40.748901363 (189h49m35.179981494s) [upload=197494]: Uploading segments... Aug 16 15:15:40.749037092 (189h49m35.180117223s) [upload=197494]: Got next segment. Inline: false Aug 16 15:15:40.749062481 (189h49m35.180142612s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:15:40.749066799 (189h49m35.18014693s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:15:42.706876320 (189h49m37.137956451s) [seg_pos={0 0}, upload=197494]: Begin upload segment... Aug 16 15:15:42.706920024 (189h49m37.138000145s) [seg_pos={0 0}, upload=197494]: Done begin upload segment. Aug 16 15:15:42.706944010 (189h49m37.138024131s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.706947897 (189h49m37.138028028s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707294150 (189h49m37.138374271s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707375695 (189h49m37.138455826s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707557744 (189h49m37.138637865s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707816520 (189h49m37.138896651s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708070395 (189h49m37.139150526s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708390919 (189h49m37.13947105s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708609336 (189h49m37.139689467s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708677527 (189h49m37.139757658s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708801684 (189h49m37.139881815s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708994624 (189h49m37.140074755s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709189947 (189h49m37.140270078s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709376594 (189h49m37.140456725s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709548043 (189h49m37.140628174s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710026518 (189h49m37.141106659s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710236450 (189h49m37.141316581s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710434188 (189h49m37.141514319s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:43.864312147 (189h49m38.295392278s) [upload=197494]: Next returned nil segment Aug 16 15:15:43.864332587 (189h49m38.295412718s) [upload=197494]: Waiting for error group managing segments... Aug 16 15:15:44.262984190 (189h49m38.694064321s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.720332763 (189h49m39.151412884s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.762855765 (189h49m39.193935886s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.888679494 (189h49m39.319759625s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.933456339 (189h49m39.36453647s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:45.030036655 (189h49m39.461116776s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.218319864 (189h49m42.649399995s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.266402625 (189h49m42.697482756s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.459775572 (189h49m42.890855703s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.757443534 (189h49m43.188523665s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:49.019151742 (189h49m43.450231873s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.176658524 (189h49m45.607738655s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.823351466 (189h49m46.254431597s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022924656 (189h49m46.454004787s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022953922 (189h49m46.454034053s) [seg_pos={0 0}, upload=197494]: Segment reached optimal threshold of 14 pieces. Aug 16 15:15:52.023339270 (189h49m46.454419401s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi): protocol: write tcp 188.166.225.170:38694->104.36.82.39:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023410757 (189h49m46.454490888s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ): protocol: rpc: dial tcp 104.36.82.58:4286: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023463979 (189h49m46.4545441s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tf9N11h9nLjVCMDpqacHE4qT6NYoPAGWLPbkLX14TZPp8dUV9): protocol: rpc: dial tcp 104.36.82.59:4308: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023513142 (189h49m46.454593273s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH): protocol: write tcp 188.166.225.170:50732->104.36.82.38:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023535665 (189h49m46.454615796s) [seg_pos={0 0}, upload=197494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:15:52.023629966 (189h49m46.454710137s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:15:52.023637009 (189h49m46.45471717s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:16:02.791852702 (189h49m57.222932823s) [upload=197494]: Done uploading segments... Aug 16 15:16:02.791875636 (189h49m57.222955757s) [upload=197494]: Upload finished. err: metaclient: pending object missing Duration: 39.68905751s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 454.690264ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:17:28 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 454.722157ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [...X....X........X.....X............X...X...X...............] duration: 20.523342112s FAIL (2025-08-16 15:17:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 20.523376599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.XXX...................................................] duration: 7.716353024s FAIL (2025-08-16 15:17:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.716370927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XXX..XXXXX...............................................] duration: 6.898760827s FAIL (2025-08-16 15:17:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.898792588s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XXX..XXXXXX...............................................] duration: 6.139748194s FAIL (2025-08-16 15:17:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.139769946s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......X.XXX................................................] duration: 765.397688ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:17:01 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 765.427655ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XXX..XXXXX..............................................] duration: 6.748895507s FAIL (2025-08-16 15:16:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.748929722s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....XX..XXX................................................] duration: 2.35737452s FAIL (2025-08-16 15:16:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.357404395s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X.XXX..................................................] duration: 7.178159944s FAIL (2025-08-16 15:16:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.178183869s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 798.399023ms FAIL (2025-08-16 15:16:48 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 798.416948ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 464.714907ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:16:45 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 464.738664ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [....X....X........X.....X............X...X...X..............] duration: 28.830363798s FAIL (2025-08-16 15:16:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 28.83038997s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X.XXX.................................................] duration: 6.851544974s FAIL (2025-08-16 15:16:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.851570503s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 774.800611ms FAIL (2025-08-16 15:16:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 774.827623ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 492.157705ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:16:08 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 492.183243ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....XXX..XXXXX.............................................] duration: 6.520589519s FAIL (2025-08-16 15:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.520640708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X.XXX................................................] duration: 7.162583227s FAIL (2025-08-16 15:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 7.162617654s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [........X.XXX...............................................] duration: 713.923567ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:16:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 713.94504ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X.X.XXX...X.......X...........X......X...X.X......X] duration: 22.700113659s FAIL (2025-08-16 15:16:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: pending object missing ======LOGS====== Aug 16 15:15:40.748806291 (189h49m35.179886422s) [upload=197494]: Starting upload Aug 16 15:15:40.748847630 (189h49m35.179927761s) [upload=197494]: Done starting upload Aug 16 15:15:40.748901363 (189h49m35.179981494s) [upload=197494]: Uploading segments... Aug 16 15:15:40.749037092 (189h49m35.180117223s) [upload=197494]: Got next segment. Inline: false Aug 16 15:15:40.749062481 (189h49m35.180142612s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:15:40.749066799 (189h49m35.18014693s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:15:42.706876320 (189h49m37.137956451s) [seg_pos={0 0}, upload=197494]: Begin upload segment... Aug 16 15:15:42.706920024 (189h49m37.138000145s) [seg_pos={0 0}, upload=197494]: Done begin upload segment. Aug 16 15:15:42.706944010 (189h49m37.138024131s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.706947897 (189h49m37.138028028s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707294150 (189h49m37.138374271s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707375695 (189h49m37.138455826s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707557744 (189h49m37.138637865s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.707816520 (189h49m37.138896651s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708070395 (189h49m37.139150526s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708390919 (189h49m37.13947105s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708609336 (189h49m37.139689467s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708677527 (189h49m37.139757658s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708801684 (189h49m37.139881815s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.708994624 (189h49m37.140074755s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709189947 (189h49m37.140270078s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709376594 (189h49m37.140456725s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.709548043 (189h49m37.140628174s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710026518 (189h49m37.141106659s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710236450 (189h49m37.141316581s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:42.710434188 (189h49m37.141514319s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Uploading piece... Aug 16 15:15:43.864312147 (189h49m38.295392278s) [upload=197494]: Next returned nil segment Aug 16 15:15:43.864332587 (189h49m38.295412718s) [upload=197494]: Waiting for error group managing segments... Aug 16 15:15:44.262984190 (189h49m38.694064321s) [noise=true , address=66.23.193.175:4560 , piece_id=QPIGCY6DEKM4NFXSAJVB4GKWDKBTM5UW2P6C6LZNKI2KTVMW5VYQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.720332763 (189h49m39.151412884s) [noise=true , address=104.36.82.35:4613 , piece_id=H7WLEQE6NCXPDZJK2UBA6DMKO5ELIZKFRA5S2XOQ52JENJCUQGDQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.762855765 (189h49m39.193935886s) [noise=true , address=79.127.201.214:4324 , piece_id=SY6G3WYMCZVBSTZSYAP3C5BBHVQ523YJHDBPHCHBBXUADEGZO44A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.888679494 (189h49m39.319759625s) [noise=true , address=79.127.201.215:4349 , piece_id=GS2N3ZE4SAIMLQRDRXZOVFEIWX63FLW7SYUH5YUR5ZUG3UX56MNA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:44.933456339 (189h49m39.36453647s) [noise=true , address=79.127.220.101:4071 , piece_id=EHDHDGY43WLQVXQUZYESCENI4HX7YFP2AISZMXS35R7V75UHNR7A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:45.030036655 (189h49m39.461116776s) [noise=true , address=79.127.201.214:4309 , piece_id=HEGEX5IVDZH4GNYV3C5FG7P4DNPQD4SC5XTUM6VLK434SW7TM2UQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.218319864 (189h49m42.649399995s) [noise=true , address=45.91.116.10:4110 , piece_id=FCJIKNCSZ4KZS53NBSYJD4NSHL3BJOZC6RLEEUTPHQR47RNFBXDA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.266402625 (189h49m42.697482756s) [noise=true , address=77.90.31.11:4575 , piece_id=SYD2OG7EJNF2OZUZURWWITUIKI5TPFVGZZFJNMKN36XJEHQR3EIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.459775572 (189h49m42.890855703s) [noise=true , address=77.90.31.21:4887 , piece_id=QTQPKQJOB7HDO3UY2YC3N2WJFAWLLUUDJUKFFXBAP7EM5QK3L6NA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:48.757443534 (189h49m43.188523665s) [noise=true , address=77.90.31.15:4709 , piece_id=QNEYRJCF6KW2U2MPSSPCPRMNCR3UJM44RSK23EKTO225L6RGJCCA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:49.019151742 (189h49m43.450231873s) [noise=true , address=104.36.82.44:4878 , piece_id=RQWZ4O3EA2KHWAILY7ZIMFROBA3MTILS7SGPYH6L6YHWW7KUEF3A, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.176658524 (189h49m45.607738655s) [noise=true , address=45.91.116.8:4049 , piece_id=L6EGJADC4APOJER33N3ULHI3L7OMLYZ2YK254AVYHQTKJBIM3LMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:51.823351466 (189h49m46.254431597s) [noise=true , address=45.91.116.12:4228 , piece_id=HTKCHM6TVHY3NXPXUYO7GN26RBLFKOICGH4OMKQNF3QGR6KETJOA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022924656 (189h49m46.454004787s) [noise=true , address=104.36.82.5:4871 , piece_id=NIE2GWXUUVM6QD7WEAOAJ5QZ5WYKXP4NY7ISXK4SBX625G3DDEAQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: <nil> Aug 16 15:15:52.022953922 (189h49m46.454034053s) [seg_pos={0 0}, upload=197494]: Segment reached optimal threshold of 14 pieces. Aug 16 15:15:52.023339270 (189h49m46.454419401s) [noise=true , address=104.36.82.39:4749 , piece_id=LXBH3WHOXHNCQCY4IRCYQZJPN7TLMTG3QZFWU47DQ2IFKVRBLGIQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi): protocol: write tcp 188.166.225.170:38694->104.36.82.39:4749: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023410757 (189h49m46.454490888s) [noise=true , address=104.36.82.58:4286 , piece_id=2I37GGHHHPMW47BNGGEU5TGMYLS2RZIFGZLNOA7NC74IUA4IE6AQ, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ): protocol: rpc: dial tcp 104.36.82.58:4286: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023463979 (189h49m46.4545441s) [noise=true , address=104.36.82.59:4308 , piece_id=PL56BNAGUJS5TAYQ6GEY53DCQOGXMFYLHDQAYEFLIIHWFDABQ2KA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tf9N11h9nLjVCMDpqacHE4qT6NYoPAGWLPbkLX14TZPp8dUV9): protocol: rpc: dial tcp 104.36.82.59:4308: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023513142 (189h49m46.454593273s) [noise=true , address=104.36.82.38:4709 , piece_id=WAGRXUYYMHQ2H6TSOO4WBLUCIKJJCKKM6ZZIZX3TXD2PTCXYDAMA, seg_pos={0 0}, upload=197494]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH): protocol: write tcp 188.166.225.170:50732->104.36.82.38:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:15:52.023535665 (189h49m46.454615796s) [seg_pos={0 0}, upload=197494]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:15:52.023629966 (189h49m46.454710137s) [upload=197494]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:15:52.023637009 (189h49m46.45471717s) [upload=197494]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:16:02.791852702 (189h49m57.222932823s) [upload=197494]: Done uploading segments... Aug 16 15:16:02.791875636 (189h49m57.222955757s) [upload=197494]: Upload finished. err: metaclient: pending object missing Duration: 22.700132265s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....X....X........X.....X............X...X...X.............] duration: 15.9989015s FAIL (2025-08-16 15:15:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 15.99892846s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XXX..XXXXXX..............................................] duration: 6.34012892s FAIL (2025-08-16 15:15:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.340148408s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......XX..XXX...............................................] duration: 2.282317492s FAIL (2025-08-16 15:15:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.282335926s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X.X.XXX...X.......X...........X......X...X.X......] duration: 35.818466257s FAIL (2025-08-16 15:15:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 276e78b0d143d0521d112d228814931449888b541b88dd132414e74b4a76fcb2 != before fba4b38cfea3bfbe838c3af72cabdfd9bc4234b46337213df35b4b0e1abd3706 ======LOGS====== Aug 16 15:10:47.840680068 (189h44m42.271760199s) [upload=197406]: Starting upload Aug 16 15:10:47.840721668 (189h44m42.271801799s) [upload=197406]: Done starting upload Aug 16 15:10:47.840817350 (189h44m42.271897481s) [upload=197406]: Uploading segments... Aug 16 15:10:47.840899337 (189h44m42.271979468s) [upload=197406]: Got next segment. Inline: false Aug 16 15:10:47.840927120 (189h44m42.272007241s) [upload=197406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:10:47.840931148 (189h44m42.272011279s) [upload=197406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:10:56.708176017 (189h44m51.139256148s) [seg_pos={0 0}, upload=197406]: Begin upload segment... Aug 16 15:10:56.708240189 (189h44m51.139320331s) [seg_pos={0 0}, upload=197406]: Done begin upload segment. Aug 16 15:10:56.708370168 (189h44m51.139450299s) [noise=true , address=104.36.82.18:4367 , piece_id=KJDRCINCOPBMCCBHSZVMAKWPAM4IVAKWWIJKFOKUN4GCOHQAMZMA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.708388544 (189h44m51.139468675s) [noise=true , address=104.36.82.60:4344 , piece_id=VBRHGZXASJD4C4QTPO5ZMRPROVWO4HRFCWUVZ6I5262J4NMKVYKQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.708889482 (189h44m51.139969613s) [noise=true , address=77.90.31.13:4660 , piece_id=L5SIFSZZJXJT7VBTWZHL5O7TFFA7YK657NXGPJHNLRCRETKURJZQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.709343150 (189h44m51.140423281s) [noise=true , address=77.90.31.12:4607 , piece_id=JVW6W6IDCN5ZAAKFRAWZLMIQXGXPYQKEWSV4FRSC77PJBKYIYTYA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.709882342 (189h44m51.140962463s) [noise=true , address=66.23.193.175:4561 , piece_id=L4ECXXEJ73GA3JWGNGV364Z3OPPUDC5JDOPEXGQPOG7DRFL5TFMQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710154492 (189h44m51.141234624s) [noise=true , address=45.91.116.13:4176 , piece_id=JNP5EPUXQCRHRVXI6LMRRHRQJZ24A4DAZASSOSFHBUPDLQYUJYEA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710333165 (189h44m51.141413286s) [noise=true , address=79.127.201.215:4349 , piece_id=3UEXIZSRFHLBH4FG2VS5QKZK6VGTVSKLIS7OIGHQFT3PSTLEXP3A, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710721608 (189h44m51.141801749s) [noise=true , address=77.90.31.14:4692 , piece_id=EBAPUXQYJ2DY72SXNXSMBVBSY73UG2L57UXHHOUHZO3VGZV7N5KQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710902423 (189h44m51.141982554s) [noise=true , address=104.36.82.5:4852 , piece_id=AE3CETZELF253OUTDT7FSTNTKTC4SQHTF5A5AGAU23ZUDJHCQIQA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.711157662 (189h44m51.142237783s) [noise=true , address=104.36.82.23:4467 , piece_id=MWWH54FTEC3V4UVBEXJRAGQYUWUAU363DAEZSO7HFLCLE4LJ7ANQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.711717383 (189h44m51.142797514s) [noise=true , address=104.36.82.44:4879 , piece_id=NKLA5IGGP5QZRA5VIJYLFVHFVIDXW6QNYUD4TI6UX5PWX6HJCVUA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712190227 (189h44m51.143270358s) [noise=true , address=104.36.82.2:4759 , piece_id=AULOIXFX4KMRNFPSQESLLSOTFHJALV3W37D6HVSUEO7GZSMXVJMQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712587238 (189h44m51.143667369s) [noise=true , address=77.90.31.18:4828 , piece_id=5SC4SCUJAUE7FNWX6FINDLGWPW7RIAWHGNORPATQO32KDJNBLIOQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712955592 (189h44m51.144035723s) [noise=true , address=104.36.82.58:4298 , piece_id=ETMLYOV6PTF2L3KO74YRN2K3HOEFSNJKV2ZHZQFD4IOGE3QQY5DA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.713400463 (189h44m51.144480594s) [noise=true , address=45.91.116.5:4321 , piece_id=2W3KEMRGV53TJYTRSI3XNBMI2AJTWYOJ6FHDV3WQZGFSQVUVH7YQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.713860433 (189h44m51.144940564s) [noise=true , address=104.36.82.8:4988 , piece_id=M7BGO5M5RPVJKKIJPWAAIIRXGZQHLROFAIO2ZCPATA3BC5HLG4MQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.714280458 (189h44m51.145360599s) [noise=true , address=77.90.31.10:4551 , piece_id=LHCDZVG6FZ624STNL2DMZAOFZUQWLLT3A2ULDQUV6I6ZVJZCAWUA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.715383298 (189h44m51.146463429s) [noise=true , address=104.36.82.36:4633 , piece_id=BCOMCIDEPMYINLJLULFXMDJBNN52SXBRZJRFNWUSL5TXZ3HCDQKQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:57.871689773 (189h44m52.302769904s) [upload=197406]: Next returned nil segment Aug 16 15:10:57.871702278 (189h44m52.302782409s) [upload=197406]: Waiting for error group managing segments... Aug 16 15:10:58.704902749 (189h44m53.13598288s) [noise=true , address=79.127.201.215:4349 , piece_id=3UEXIZSRFHLBH4FG2VS5QKZK6VGTVSKLIS7OIGHQFT3PSTLEXP3A, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.731855890 (189h44m53.162936021s) [noise=true , address=104.36.82.2:4759 , piece_id=AULOIXFX4KMRNFPSQESLLSOTFHJALV3W37D6HVSUEO7GZSMXVJMQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.766776807 (189h44m53.197856938s) [noise=true , address=104.36.82.23:4467 , piece_id=MWWH54FTEC3V4UVBEXJRAGQYUWUAU363DAEZSO7HFLCLE4LJ7ANQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.789205572 (189h44m53.220285703s) [noise=true , address=104.36.82.44:4879 , piece_id=NKLA5IGGP5QZRA5VIJYLFVHFVIDXW6QNYUD4TI6UX5PWX6HJCVUA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.813758261 (189h44m53.244838392s) [noise=true , address=104.36.82.5:4852 , piece_id=AE3CETZELF253OUTDT7FSTNTKTC4SQHTF5A5AGAU23ZUDJHCQIQA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.053968485 (189h44m53.485048616s) [noise=true , address=104.36.82.18:4367 , piece_id=KJDRCINCOPBMCCBHSZVMAKWPAM4IVAKWWIJKFOKUN4GCOHQAMZMA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.159179323 (189h44m53.590259494s) [noise=true , address=104.36.82.60:4344 , piece_id=VBRHGZXASJD4C4QTPO5ZMRPROVWO4HRFCWUVZ6I5262J4NMKVYKQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.301656827 (189h44m53.732736988s) [noise=true , address=104.36.82.36:4633 , piece_id=BCOMCIDEPMYINLJLULFXMDJBNN52SXBRZJRFNWUSL5TXZ3HCDQKQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.584222744 (189h44m54.015302865s) [noise=true , address=77.90.31.12:4607 , piece_id=JVW6W6IDCN5ZAAKFRAWZLMIQXGXPYQKEWSV4FRSC77PJBKYIYTYA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.702996703 (189h44m54.134076844s) [noise=true , address=77.90.31.18:4828 , piece_id=5SC4SCUJAUE7FNWX6FINDLGWPW7RIAWHGNORPATQO32KDJNBLIOQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.754201017 (189h44m54.185281148s) [noise=true , address=77.90.31.14:4692 , piece_id=EBAPUXQYJ2DY72SXNXSMBVBSY73UG2L57UXHHOUHZO3VGZV7N5KQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.867688142 (189h44m54.298768273s) [noise=true , address=66.23.193.175:4561 , piece_id=L4ECXXEJ73GA3JWGNGV364Z3OPPUDC5JDOPEXGQPOG7DRFL5TFMQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.936118284 (189h44m54.367198416s) [noise=true , address=45.91.116.5:4321 , piece_id=2W3KEMRGV53TJYTRSI3XNBMI2AJTWYOJ6FHDV3WQZGFSQVUVH7YQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:11:00.456960055 (189h44m54.888040186s) [noise=true , address=77.90.31.10:4551 , piece_id=LHCDZVG6FZ624STNL2DMZAOFZUQWLLT3A2ULDQUV6I6ZVJZCAWUA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:11:00.456997666 (189h44m54.888077837s) [seg_pos={0 0}, upload=197406]: Segment reached optimal threshold of 14 pieces. Aug 16 15:11:00.457135861 (189h44m54.888216042s) [noise=true , address=45.91.116.13:4176 , piece_id=JNP5EPUXQCRHRVXI6LMRRHRQJZ24A4DAZASSOSFHBUPDLQYUJYEA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Aug 16 15:11:00.457199041 (189h44m54.888279172s) [noise=true , address=104.36.82.8:4988 , piece_id=M7BGO5M5RPVJKKIJPWAAIIRXGZQHLROFAIO2ZCPATA3BC5HLG4MQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qs9EDisetfkwcMS8r1d1twWZoTzaz2QZAo2aJakM53ZyJs2Je): protocol: context canceled; context canceled Aug 16 15:11:00.457256441 (189h44m54.888336562s) [noise=true , address=77.90.31.13:4660 , piece_id=L5SIFSZZJXJT7VBTWZHL5O7TFFA7YK657NXGPJHNLRCRETKURJZQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG): protocol: expected piece hash; context canceled; EOF Aug 16 15:11:00.457325554 (189h44m54.888405685s) [noise=true , address=104.36.82.58:4298 , piece_id=ETMLYOV6PTF2L3KO74YRN2K3HOEFSNJKV2ZHZQFD4IOGE3QQY5DA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MWsyGhKkDhjngUDDRvxmhQddbNnRZ1A3mpzEtjLs9ZcRC5LBz): protocol: read tcp 188.166.225.170:53450->104.36.82.58:4298: use of closed network connection; context canceled Aug 16 15:11:00.457347436 (189h44m54.888427567s) [seg_pos={0 0}, upload=197406]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:11:00.457409865 (189h44m54.888489996s) [upload=197406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:11:00.457414173 (189h44m54.888494304s) [upload=197406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:11:07.319008928 (189h45m1.750089059s) [upload=197406]: Done uploading segments... Aug 16 15:11:07.319061679 (189h45m1.75014181s) [upload=197406]: Upload finished. err: <nil> Duration: 35.818490202s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X.XXX...............................................] duration: 6.892672626s FAIL (2025-08-16 15:15:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.892699967s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 753.731876ms FAIL (2025-08-16 15:15:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 753.753597ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......XXX..XXXXX............................................] duration: 8.190549792s FAIL (2025-08-16 15:15:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.190570903s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XXX..XXXXXX.............................................] duration: 6.18544936s FAIL (2025-08-16 15:15:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.18547064s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 489.663353ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 15:15:13 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 489.710112ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [......X....X........X.....X............X...X...X............] duration: 12.173526321s FAIL (2025-08-16 15:15:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 12.173550038s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......XX..XXX..............................................] duration: 2.365023816s FAIL (2025-08-16 15:15:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.36504629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........X.X.XXX...X.......X...........X......X...X.X.....] duration: 43.61761994s FAIL (2025-08-16 15:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 276e78b0d143d0521d112d228814931449888b541b88dd132414e74b4a76fcb2 != before fba4b38cfea3bfbe838c3af72cabdfd9bc4234b46337213df35b4b0e1abd3706 ======LOGS====== Aug 16 15:10:47.840680068 (189h44m42.271760199s) [upload=197406]: Starting upload Aug 16 15:10:47.840721668 (189h44m42.271801799s) [upload=197406]: Done starting upload Aug 16 15:10:47.840817350 (189h44m42.271897481s) [upload=197406]: Uploading segments... Aug 16 15:10:47.840899337 (189h44m42.271979468s) [upload=197406]: Got next segment. Inline: false Aug 16 15:10:47.840927120 (189h44m42.272007241s) [upload=197406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:10:47.840931148 (189h44m42.272011279s) [upload=197406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:10:56.708176017 (189h44m51.139256148s) [seg_pos={0 0}, upload=197406]: Begin upload segment... Aug 16 15:10:56.708240189 (189h44m51.139320331s) [seg_pos={0 0}, upload=197406]: Done begin upload segment. Aug 16 15:10:56.708370168 (189h44m51.139450299s) [noise=true , address=104.36.82.18:4367 , piece_id=KJDRCINCOPBMCCBHSZVMAKWPAM4IVAKWWIJKFOKUN4GCOHQAMZMA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.708388544 (189h44m51.139468675s) [noise=true , address=104.36.82.60:4344 , piece_id=VBRHGZXASJD4C4QTPO5ZMRPROVWO4HRFCWUVZ6I5262J4NMKVYKQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.708889482 (189h44m51.139969613s) [noise=true , address=77.90.31.13:4660 , piece_id=L5SIFSZZJXJT7VBTWZHL5O7TFFA7YK657NXGPJHNLRCRETKURJZQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.709343150 (189h44m51.140423281s) [noise=true , address=77.90.31.12:4607 , piece_id=JVW6W6IDCN5ZAAKFRAWZLMIQXGXPYQKEWSV4FRSC77PJBKYIYTYA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.709882342 (189h44m51.140962463s) [noise=true , address=66.23.193.175:4561 , piece_id=L4ECXXEJ73GA3JWGNGV364Z3OPPUDC5JDOPEXGQPOG7DRFL5TFMQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710154492 (189h44m51.141234624s) [noise=true , address=45.91.116.13:4176 , piece_id=JNP5EPUXQCRHRVXI6LMRRHRQJZ24A4DAZASSOSFHBUPDLQYUJYEA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710333165 (189h44m51.141413286s) [noise=true , address=79.127.201.215:4349 , piece_id=3UEXIZSRFHLBH4FG2VS5QKZK6VGTVSKLIS7OIGHQFT3PSTLEXP3A, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710721608 (189h44m51.141801749s) [noise=true , address=77.90.31.14:4692 , piece_id=EBAPUXQYJ2DY72SXNXSMBVBSY73UG2L57UXHHOUHZO3VGZV7N5KQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.710902423 (189h44m51.141982554s) [noise=true , address=104.36.82.5:4852 , piece_id=AE3CETZELF253OUTDT7FSTNTKTC4SQHTF5A5AGAU23ZUDJHCQIQA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.711157662 (189h44m51.142237783s) [noise=true , address=104.36.82.23:4467 , piece_id=MWWH54FTEC3V4UVBEXJRAGQYUWUAU363DAEZSO7HFLCLE4LJ7ANQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.711717383 (189h44m51.142797514s) [noise=true , address=104.36.82.44:4879 , piece_id=NKLA5IGGP5QZRA5VIJYLFVHFVIDXW6QNYUD4TI6UX5PWX6HJCVUA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712190227 (189h44m51.143270358s) [noise=true , address=104.36.82.2:4759 , piece_id=AULOIXFX4KMRNFPSQESLLSOTFHJALV3W37D6HVSUEO7GZSMXVJMQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712587238 (189h44m51.143667369s) [noise=true , address=77.90.31.18:4828 , piece_id=5SC4SCUJAUE7FNWX6FINDLGWPW7RIAWHGNORPATQO32KDJNBLIOQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.712955592 (189h44m51.144035723s) [noise=true , address=104.36.82.58:4298 , piece_id=ETMLYOV6PTF2L3KO74YRN2K3HOEFSNJKV2ZHZQFD4IOGE3QQY5DA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.713400463 (189h44m51.144480594s) [noise=true , address=45.91.116.5:4321 , piece_id=2W3KEMRGV53TJYTRSI3XNBMI2AJTWYOJ6FHDV3WQZGFSQVUVH7YQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.713860433 (189h44m51.144940564s) [noise=true , address=104.36.82.8:4988 , piece_id=M7BGO5M5RPVJKKIJPWAAIIRXGZQHLROFAIO2ZCPATA3BC5HLG4MQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.714280458 (189h44m51.145360599s) [noise=true , address=77.90.31.10:4551 , piece_id=LHCDZVG6FZ624STNL2DMZAOFZUQWLLT3A2ULDQUV6I6ZVJZCAWUA, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:56.715383298 (189h44m51.146463429s) [noise=true , address=104.36.82.36:4633 , piece_id=BCOMCIDEPMYINLJLULFXMDJBNN52SXBRZJRFNWUSL5TXZ3HCDQKQ, seg_pos={0 0}, upload=197406]: Uploading piece... Aug 16 15:10:57.871689773 (189h44m52.302769904s) [upload=197406]: Next returned nil segment Aug 16 15:10:57.871702278 (189h44m52.302782409s) [upload=197406]: Waiting for error group managing segments... Aug 16 15:10:58.704902749 (189h44m53.13598288s) [noise=true , address=79.127.201.215:4349 , piece_id=3UEXIZSRFHLBH4FG2VS5QKZK6VGTVSKLIS7OIGHQFT3PSTLEXP3A, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.731855890 (189h44m53.162936021s) [noise=true , address=104.36.82.2:4759 , piece_id=AULOIXFX4KMRNFPSQESLLSOTFHJALV3W37D6HVSUEO7GZSMXVJMQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.766776807 (189h44m53.197856938s) [noise=true , address=104.36.82.23:4467 , piece_id=MWWH54FTEC3V4UVBEXJRAGQYUWUAU363DAEZSO7HFLCLE4LJ7ANQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.789205572 (189h44m53.220285703s) [noise=true , address=104.36.82.44:4879 , piece_id=NKLA5IGGP5QZRA5VIJYLFVHFVIDXW6QNYUD4TI6UX5PWX6HJCVUA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:58.813758261 (189h44m53.244838392s) [noise=true , address=104.36.82.5:4852 , piece_id=AE3CETZELF253OUTDT7FSTNTKTC4SQHTF5A5AGAU23ZUDJHCQIQA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.053968485 (189h44m53.485048616s) [noise=true , address=104.36.82.18:4367 , piece_id=KJDRCINCOPBMCCBHSZVMAKWPAM4IVAKWWIJKFOKUN4GCOHQAMZMA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.159179323 (189h44m53.590259494s) [noise=true , address=104.36.82.60:4344 , piece_id=VBRHGZXASJD4C4QTPO5ZMRPROVWO4HRFCWUVZ6I5262J4NMKVYKQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.301656827 (189h44m53.732736988s) [noise=true , address=104.36.82.36:4633 , piece_id=BCOMCIDEPMYINLJLULFXMDJBNN52SXBRZJRFNWUSL5TXZ3HCDQKQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.584222744 (189h44m54.015302865s) [noise=true , address=77.90.31.12:4607 , piece_id=JVW6W6IDCN5ZAAKFRAWZLMIQXGXPYQKEWSV4FRSC77PJBKYIYTYA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.702996703 (189h44m54.134076844s) [noise=true , address=77.90.31.18:4828 , piece_id=5SC4SCUJAUE7FNWX6FINDLGWPW7RIAWHGNORPATQO32KDJNBLIOQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.754201017 (189h44m54.185281148s) [noise=true , address=77.90.31.14:4692 , piece_id=EBAPUXQYJ2DY72SXNXSMBVBSY73UG2L57UXHHOUHZO3VGZV7N5KQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.867688142 (189h44m54.298768273s) [noise=true , address=66.23.193.175:4561 , piece_id=L4ECXXEJ73GA3JWGNGV364Z3OPPUDC5JDOPEXGQPOG7DRFL5TFMQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:10:59.936118284 (189h44m54.367198416s) [noise=true , address=45.91.116.5:4321 , piece_id=2W3KEMRGV53TJYTRSI3XNBMI2AJTWYOJ6FHDV3WQZGFSQVUVH7YQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:11:00.456960055 (189h44m54.888040186s) [noise=true , address=77.90.31.10:4551 , piece_id=LHCDZVG6FZ624STNL2DMZAOFZUQWLLT3A2ULDQUV6I6ZVJZCAWUA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: <nil> Aug 16 15:11:00.456997666 (189h44m54.888077837s) [seg_pos={0 0}, upload=197406]: Segment reached optimal threshold of 14 pieces. Aug 16 15:11:00.457135861 (189h44m54.888216042s) [noise=true , address=45.91.116.13:4176 , piece_id=JNP5EPUXQCRHRVXI6LMRRHRQJZ24A4DAZASSOSFHBUPDLQYUJYEA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Aug 16 15:11:00.457199041 (189h44m54.888279172s) [noise=true , address=104.36.82.8:4988 , piece_id=M7BGO5M5RPVJKKIJPWAAIIRXGZQHLROFAIO2ZCPATA3BC5HLG4MQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qs9EDisetfkwcMS8r1d1twWZoTzaz2QZAo2aJakM53ZyJs2Je): protocol: context canceled; context canceled Aug 16 15:11:00.457256441 (189h44m54.888336562s) [noise=true , address=77.90.31.13:4660 , piece_id=L5SIFSZZJXJT7VBTWZHL5O7TFFA7YK657NXGPJHNLRCRETKURJZQ, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG): protocol: expected piece hash; context canceled; EOF Aug 16 15:11:00.457325554 (189h44m54.888405685s) [noise=true , address=104.36.82.58:4298 , piece_id=ETMLYOV6PTF2L3KO74YRN2K3HOEFSNJKV2ZHZQFD4IOGE3QQY5DA, seg_pos={0 0}, upload=197406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MWsyGhKkDhjngUDDRvxmhQddbNnRZ1A3mpzEtjLs9ZcRC5LBz): protocol: read tcp 188.166.225.170:53450->104.36.82.58:4298: use of closed network connection; context canceled Aug 16 15:11:00.457347436 (189h44m54.888427567s) [seg_pos={0 0}, upload=197406]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:11:00.457409865 (189h44m54.888489996s) [upload=197406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:11:00.457414173 (189h44m54.888494304s) [upload=197406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:11:07.319008928 (189h45m1.750089059s) [upload=197406]: Done uploading segments... Aug 16 15:11:07.319061679 (189h45m1.75014181s) [upload=197406]: Upload finished. err: <nil> Duration: 43.617644989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.........X.XXX..............................................] duration: 782.074669ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 15:14:49 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 782.0961ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X.XXX..............................................] duration: 15.66003969s FAIL (2025-08-16 15:14:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 15.660061081s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 765.847878ms FAIL (2025-08-16 15:14:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 765.866265ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XXX..XXXXXX............................................] duration: 6.735282735s FAIL (2025-08-16 15:14:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.735296682s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 739.045486ms FAIL (2025-08-16 15:14:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 739.069954ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [.......X....X........X.....X............X...X...X...........] duration: 8.358711871s FAIL (2025-08-16 15:14:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a76d66aa20905260742a3b4f560d23a09b1c5b91047b45673d8930e17b064963 != before 4160bb81ffbc32d6541fbf6fba1b567fe8ac819affe89325c3a39446eb6aeca8 ======LOGS====== Aug 16 15:07:22.607289776 (189h41m17.038369907s) [upload=197353]: Starting upload Aug 16 15:07:22.607331335 (189h41m17.038411456s) [upload=197353]: Done starting upload Aug 16 15:07:22.607400288 (189h41m17.038480419s) [upload=197353]: Uploading segments... Aug 16 15:07:22.607440354 (189h41m17.038520485s) [upload=197353]: Got next segment. Inline: true Aug 16 15:07:22.607443490 (189h41m17.038523621s) [upload=197353]: Waiting for error group managing segments... Aug 16 15:07:22.607470211 (189h41m17.038550332s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:07:22.607473988 (189h41m17.038554119s) [upload=197353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:07:22.607476713 (189h41m17.038556844s) [upload=197353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:07:44.457789165 (189h41m38.888869296s) [upload=197353]: Done uploading segments... Aug 16 15:07:44.457821888 (189h41m38.888902019s) [upload=197353]: Upload finished. err: <nil> Duration:* 8.358734002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.X.XXX.............................................] duration: 9.108565395s FAIL (2025-08-16 15:14:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 9.108595995s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)