Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X.................................X....X..........X...] duration: 1.888568257s FAIL (2025-07-27 01:08:05 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.888592223s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X...................] duration: 5.904353251s FAIL (2025-07-27 01:08:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90437383s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X..X...XX..............................................] duration: 6.536398961s FAIL (2025-07-27 01:08:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.536421593s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.....X...XXX..XXX..X..X.X..............X....................] duration: 5.677845223s FAIL (2025-07-27 01:07:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.677872766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX.................] duration: 5.548014225s FAIL (2025-07-27 01:07:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.54804334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X..X...XX.............................................] duration: 5.876455011s FAIL (2025-07-27 01:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.876478277s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......X.................................X....X..........X..] duration: 1.894727076s FAIL (2025-07-27 01:07:30 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.894752645s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX...............] duration: 6.146912034s FAIL (2025-07-27 01:07:22 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.146940629s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X..X...XX............................................] duration: 6.20722392s FAIL (2025-07-27 01:07:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.207244429s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX................] duration: 5.899047634s FAIL (2025-07-27 01:07:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.899069504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........X.................................X....X..........X.] duration: 1.858901523s FAIL (2025-07-27 01:07:03 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.858925629s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX..............] duration: 5.703724389s FAIL (2025-07-27 01:06:54 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.703756702s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X..................] duration: 5.889020525s FAIL (2025-07-27 01:06:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.889043699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [......X...XXX..XXX..X..X.X..............X...................] duration: 5.747149705s FAIL (2025-07-27 01:06:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.747172148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX...............] duration: 6.74636584s FAIL (2025-07-27 01:06:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.746395756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X.................] duration: 5.820143785s FAIL (2025-07-27 01:06:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.820173792s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X..X...XX...........................................] duration: 6.164135648s FAIL (2025-07-27 01:06:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.164157911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX.............] duration: 6.807591789s FAIL (2025-07-27 01:06:14 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.807615064s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX..............] duration: 9.482955054s FAIL (2025-07-27 01:06:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.482991855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........X.................................X....X..........X] duration: 1.647326795s FAIL (2025-07-27 01:05:58 UTC) orbiter warning: digest mismatch: after 0fab96406578ff4f6e5ac3bcbf27e9ffb66f17d6676b38ac64e45ce1eebe95f6 != before 0f0419772558bb804c8dac0f8b10ef5a33d3e2f1a02a6ab2fe9f121ad674c5d9 Duration: 1.647350732s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX............] duration: 5.72240737s FAIL (2025-07-27 01:05:43 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.722433639s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X......X...XXX..XXX..X..X.X..............X..................] duration: 5.481914503s FAIL (2025-07-27 01:05:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.482113364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........X.................................X....X..........] duration: 1.904617757s FAIL (2025-07-27 01:05:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.90464517s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X..X...XX..........................................] duration: 5.920448906s FAIL (2025-07-27 01:05:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.920470727s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X................] duration: 7.150054925s FAIL (2025-07-27 01:05:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.150087075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX.............] duration: 5.590025586s FAIL (2025-07-27 01:04:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.590046576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X..X...XX.........................................] duration: 5.890223846s FAIL (2025-07-27 01:04:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890249947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X.................................X....X.........] duration: 1.936395694s FAIL (2025-07-27 01:04:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.936422444s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX...........] duration: 16.299929669s FAIL (2025-07-27 01:04:44 UTC) orbiter warning: uplink: context deadline exceeded Duration: 16.299952734s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X...............] duration: 12.308028315s FAIL (2025-07-27 01:04:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.308052952s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XX......X...XXX..XXX..X..X.X..............X.................] duration: 5.879569462s FAIL (2025-07-27 01:04:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.879591295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XX......X...XXX..XXX..X..X.X..............X................] duration: 5.706623307s FAIL (2025-07-27 01:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.706660739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX............] duration: 11.511666978s FAIL (2025-07-27 01:04:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 11.51169412s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X.................................X....X........] duration: 1.887520675s FAIL (2025-07-27 01:04:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.887538558s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X.................................X....X.......] duration: 1.896449068s FAIL (2025-07-27 01:03:50 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.89647615s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X..X...XX........................................] duration: 5.967174506s FAIL (2025-07-27 01:03:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.967194965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X..............] duration: 5.665392769s FAIL (2025-07-27 01:03:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.665418368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..XX......X...XXX..XXX..X..X.X..............X...............] duration: 7.811139043s FAIL (2025-07-27 01:03:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.811167498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX..........] duration: 6.284232087s FAIL (2025-07-27 01:03:25 UTC) orbiter warning: uplink: context deadline exceeded Duration: 6.284258687s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...XX......X...XXX..XXX..X..X.X..............X..............] duration: 5.583534504s FAIL (2025-07-27 01:03:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.58355928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X.............] duration: 5.849702195s FAIL (2025-07-27 01:02:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.849724257s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X..X...XX.......................................] duration: 6.510964638s FAIL (2025-07-27 01:02:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.510988265s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX...........] duration: 5.772641157s FAIL (2025-07-27 01:02:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.77266931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X.................................X....X......] duration: 2.188066063s FAIL (2025-07-27 01:02:48 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.188096903s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......XXXXXX..........] duration: 13.106701053s FAIL (2025-07-27 01:02:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== false Jul 27 00:54:57.821101936 (61h19m7.543664977s) [upload=58793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:57.821108168 (61h19m7.543671208s) [upload=58793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:58.268999381 (61h19m7.991562411s) [seg_pos={0 0}, upload=58793]: Begin upload segment... Jul 27 00:54:58.269047784 (61h19m7.991610824s) [seg_pos={0 0}, upload=58793]: Done begin upload segment. Jul 27 00:54:58.269068714 (61h19m7.991631754s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269108290 (61h19m7.99167133s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269416039 (61h19m7.991979079s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269588800 (61h19m7.99215184s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269666007 (61h19m7.992229048s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.269938439 (61h19m7.992501479s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270186764 (61h19m7.992749794s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270456489 (61h19m7.993019529s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270693122 (61h19m7.993256162s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270924895 (61h19m7.993487935s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.270973609 (61h19m7.993536649s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271180806 (61h19m7.993743846s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271367583 (61h19m7.993930613s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271609946 (61h19m7.994172986s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271884882 (61h19m7.994447922s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.271925130 (61h19m7.99448817s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272230464 (61h19m7.994793504s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:58.272352857 (61h19m7.994915898s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Uploading piece... Jul 27 00:54:59.403085511 (61h19m9.125648551s) [upload=58793]: Next returned nil segment Jul 27 00:54:59.403106221 (61h19m9.125669281s) [upload=58793]: Waiting for error group managing segments... Jul 27 00:54:59.699317478 (61h19m9.421880518s) [noise=true , address=64.49.13.137:4502 , piece_id=COTTEJEOI7INJS3ZDBHS2CLIGMREMHZGHN7MB22TJR2Y4LZMVRRA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.712155049 (61h19m9.434718079s) [noise=true , address=79.127.205.235:4682 , piece_id=4BXTF4IATE3AD46376DXSOM25E3UL75T4DYKSI7YKKAY4XDUGXZQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:54:59.718874015 (61h19m9.441437055s) [noise=true , address=79.127.205.245:4242 , piece_id=MH52WQ77OW4ZDQFVVFICHDWERB3WGGZXONOOA66J4P32ILZAREQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.219903397 (61h19m9.942466437s) [noise=true , address=77.90.31.16:4751 , piece_id=XC46QZVCMKTVTLQYDPGM4ZRRD4KDIFCJTQMHFDMY633FQG74FWWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:00.393745135 (61h19m10.116308165s) [noise=true , address=77.90.31.11:4573 , piece_id=352UN4WWJYSEOQ4VR4CDGD5XWCDY6TXNXZJW6COACYDDPNTGZSOA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.589079921 (61h19m11.311642972s) [noise=true , address=79.127.219.47:4072 , piece_id=JBTVVQE3YIPDNKFAWIPDB5UBQZXP7KJJW4OBCXTAZDUQWLODTZWQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:01.684825540 (61h19m11.40738858s) [noise=true , address=79.127.220.101:4041 , piece_id=DBBBRJX5NPIPW5P72KSEANTJRZEM5MMUJQQTLXDSCYVUQY6IIIXA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:02.296389006 (61h19m12.018952046s) [noise=true , address=156.146.43.225:4448 , piece_id=TAPXLBQNT5ERYAWMWFBHQF4LYXAZJS4XXIAHXT57S3ORLKEX4MQQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.015602295 (61h19m12.738165335s) [noise=true , address=79.127.219.38:4001 , piece_id=ZKBCPE6IA6NQJDRGDGFQT2NNGIXAH64ZZ27OE2WPUR54XOQQ2Q3Q, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:03.588380603 (61h19m13.310943643s) [noise=true , address=77.90.31.10:4525 , piece_id=B7APUOSESFZRV2QUHZ6KZLR7VRJTPHNZ3FFWMEPYV6O7I7EQFVDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:07.419707501 (61h19m17.142270581s) [noise=true , address=79.127.201.215:4344 , piece_id=BFA6WIJYTUHSVLKQVLXWCBKNYJJB6CXUUVTZMZEESHH4QG3OV4NQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:10.064731586 (61h19m19.787294626s) [noise=true , address=64.49.13.140:4514 , piece_id=RXOR5I3R3HF6WUJPKES4OQFTPMETGOEFPZB2KTUS4WVSQUIQQTDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: <nil> Jul 27 00:55:57.170253087 (61h20m6.892816127s) [noise=true , address=104.36.82.53:4153 , piece_id=S2YS2I44RPQEKILNXGKZ6JKUWZIU47ZJTQTDDH6IKCKCEL7IM2NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: write tcp 188.166.225.170:41520->104.36.82.53:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170329522 (61h20m6.892892562s) [noise=true , address=104.36.82.47:4545 , piece_id=F4LGWDTZ3CMZIZBDGZZ4TGZ6GKFYE3GNW2B6SZEXBBMYYFS6Y3CQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170370181 (61h20m6.892933221s) [noise=true , address=104.36.82.52:4091 , piece_id=6Z4ZKBKTEMXCSMNIC6277XZMSJPBKTFW5RJ2PC726VVJTALRJ6NA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170401270 (61h20m6.89296431s) [noise=true , address=104.36.82.46:4500 , piece_id=SQPQJFK3QRENUZNGARKNCCVDHDY2ZXIZNVOFZV76AC63YFACQTBQ, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: rpc: dial tcp 104.36.82.46:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170514156 (61h20m6.893077196s) [noise=true , address=104.36.82.49:4005 , piece_id=J6YISXBRJWWSOJLDNE4U72EQFENIS7SPYZM4SCEVJG2SUGDHRNDA, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:104.36.82.49:4005): protocol: rpc: dial tcp 104.36.82.49:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170524104 (61h20m6.893087155s) [noise=true , address=104.36.82.15:4289 , piece_id=MFZHE2Q766VXNQ43QOIPXO33RASSYDVTJ2QLTVR4ACCHMGJLVV2A, seg_pos={0 0}, upload=58793]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:55:57.170556958 (61h20m6.893119998s) [seg_pos={0 0}, upload=58793]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:55:57.400256958 (61h20m7.122819998s) [upload=58793]: Done uploading segments... Jul 27 00:55:57.400281876 (61h20m7.122844916s) [upload=58793]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.106727093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXX.X.XX..XXXXX.XXXX..XXXX.....X....X....XXX.........] duration: 5.830555284s FAIL (2025-07-27 01:02:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 5.830580152s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..XXXXX.X.XXXXXXXXX.XXXXXXXX..X.X.X..........X............] duration: 6.061691153s FAIL (2025-07-27 01:02:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7326QE72JPEXPXFHPPYHSNKIURIMRGIQ, seg_pos={0 0}, upload=58778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:33562->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:03.385284649 (61h18m13.107847679s) [seg_pos={0 0}, upload=58778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:54:03.385367337 (61h18m13.107930377s) [upload=58778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:54:03.385371374 (61h18m13.107934414s) [upload=58778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:54:03.688084400 (61h18m13.41064744s) [upload=58778]: Done uploading segments... Jul 27 00:54:03.688101954 (61h18m13.410664994s) [upload=58778]: Upload finished. err: <nil> Jul 27 00:54:29.969393323 (61h18m39.691956364s) [upload=58786]: Starting upload Jul 27 00:54:29.969484158 (61h18m39.692047198s) [upload=58786]: Done starting upload Jul 27 00:54:29.969609477 (61h18m39.692172517s) [upload=58786]: Uploading segments... Jul 27 00:54:29.969791546 (61h18m39.692354576s) [upload=58786]: Got next segment. Inline: false Jul 27 00:54:29.969897859 (61h18m39.6924609s) [upload=58786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:54:29.969903801 (61h18m39.692466841s) [upload=58786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:54:30.202493651 (61h18m39.925056691s) [seg_pos={0 0}, upload=58786]: Begin upload segment... Jul 27 00:54:30.202544298 (61h18m39.925107338s) [seg_pos={0 0}, upload=58786]: Done begin upload segment. Jul 27 00:54:30.202565960 (61h18m39.92512899s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202629651 (61h18m39.925192691s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.202926890 (61h18m39.92548992s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203102106 (61h18m39.925665146s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203204351 (61h18m39.925767391s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203445583 (61h18m39.926008623s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203491891 (61h18m39.926054921s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.203718445 (61h18m39.926281475s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204010243 (61h18m39.926573283s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204257396 (61h18m39.926820436s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.204507925 (61h18m39.927070976s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205281365 (61h18m39.927844435s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205688314 (61h18m39.928251354s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.205981905 (61h18m39.928544945s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.206218718 (61h18m39.928781759s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:30.207082942 (61h18m39.929645982s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Uploading piece... Jul 27 00:54:31.892857589 (61h18m41.615420639s) [upload=58786]: Next returned nil segment Jul 27 00:54:31.892877968 (61h18m41.615441008s) [upload=58786]: Waiting for error group managing segments... Jul 27 00:54:32.223482398 (61h18m41.946045438s) [noise=true , address=45.91.116.6:4974 , piece_id=LR3SRAZFX3NSX7KTZOR643UHIIJSXI4XTBCXYA6W2SFKSEBN7Q2Q, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.237060637 (61h18m41.959623677s) [noise=true , address=45.91.116.4:4939 , piece_id=ZR6IIF5N5GSW6XIMVVRFVAE56M4HNLL5J7SSJE5M5UUFXKZFJSFQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.299221539 (61h18m42.021784619s) [noise=true , address=45.91.116.7:4009 , piece_id=4V2UC4QPMJKHBAI7BVHTVPZQFY667XVV3DCXAN3GKL444IHXEVEA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325716521 (61h18m42.048279561s) [noise=true , address=45.91.116.6:4973 , piece_id=VKQD7SENX3EDE34BXHGL4Z7MTET2OWSFLTOZ4PPYTDD2QVUHZEJQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.325884473 (61h18m42.048447503s) [noise=true , address=45.91.116.12:4227 , piece_id=J2OI4W4CT2FLAUB7WOGOAMTRWRS4PB5AQXLF4XFZNNO2J6LL46SA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.372983466 (61h18m42.095546506s) [noise=true , address=45.91.116.5:4299 , piece_id=5K6GYVCVQ46VZRUMNZ3PEITFO66VABEGLAV7NZCV5M36QVUVMFCA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.379844103 (61h18m42.102407133s) [noise=true , address=45.91.116.7:4010 , piece_id=EKECSPS64QHNIO3EIRAVAT35KA5TCPBMDBLT5TSB3LZWKP6KZZHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.384666850 (61h18m42.10722989s) [noise=true , address=77.90.31.14:4662 , piece_id=VBVLHQKNOIU25REDYLILDDF3XRZFFU5W5IY4I7YMUE2WTZXLBHEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:32.472371113 (61h18m42.194934143s) [noise=true , address=77.90.31.16:4735 , piece_id=N32CSF74OC4RGUPMJST5IV5P3FI6RBEUHKXCPXVNUFNHHSN5ZH4A, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:33.667049984 (61h18m43.389613024s) [noise=true , address=77.90.31.19:4843 , piece_id=KHZD4RVLBISBTIER5RAE2UCFCEFYYWCPMMXLSKIRUTIQTMLODQXQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:34.213712174 (61h18m43.936275214s) [noise=true , address=77.90.31.12:4592 , piece_id=EA7S4T5VCRCJBRETDY6FJ43SOP7NWEVMR2F6L64J3TB6TNOOCKAA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:42.252927986 (61h18m51.975491026s) [noise=true , address=77.90.31.10:4520 , piece_id=4GRJDJRCNJPKJ5VV23VRZRNBLAOCJEHFVW534O3M3TQ4SRSP2FEQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: <nil> Jul 27 00:54:59.328639826 (61h19m9.051202866s) [noise=true , address=104.36.82.5:4838 , piece_id=JNEB4FUOK534MWNRW4DB5VJPPCUIBDA6FIX2YXWSRXNFQ6GDACHA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328713146 (61h19m9.051276186s) [noise=true , address=104.36.82.7:4910 , piece_id=ZJYKL4SR6BZP6OYD7QQOZADAR7GDVKVXV4GERU6FQE52RXZGCGOA, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.328897458 (61h19m9.051460488s) [noise=true , address=104.36.82.15:4259 , piece_id=2CYXGFMVV7Z4PMRHEJ23VWSK53BM3C4DRQPNSPOI55MY7FLS2OSQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:42398->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329496955 (61h19m9.052059995s) [noise=true , address=104.36.82.9:4005 , piece_id=3ARJN3YOZQ7Y3YYCSDQOHV5PEWB7WWJSS3WCHLJ52KWKD32CUKDQ, seg_pos={0 0}, upload=58786]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:54:59.329528055 (61h19m9.052091095s) [seg_pos={0 0}, upload=58786]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:54:59.557689493 (61h19m9.280252533s) [upload=58786]: Done uploading segments... Jul 27 00:54:59.557710713 (61h19m9.280273753s) [upload=58786]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.061722392s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [....XX......X...XXX..XXX..X..X.X..............X.............] duration: 5.644230303s FAIL (2025-07-27 01:02:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.644252995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X.................................X....X.....] duration: 1.927529824s FAIL (2025-07-27 01:02:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.9275489s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..X..X...XX......................................] duration: 6.408048323s FAIL (2025-07-27 01:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.408068743s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)